macOS Can't authenticate using SSH key when running script via launchd

[daveh0]

I have a Bash script that executes an rsync command to sync files with a remote server.

#!/bin/bash
rsync -auvzP  --exclude=.bundle --exclude=node_modules --exclude=tmp  '/Volumes/Norman Data/me/.bash_profile' '/Volumes/Norman Data/me/Documents' --exclude=remote me@example.com:backup/

This works when I run it from the terminal as I have ssh key authentication set up for my user at example.com. However, when launchd calls this script, I get the following error indicating that it cannot find my user's key file or it is invalid:

Permission denied, please try again.
Permission denied, please try again.
me@example.com: Permission denied (publickey,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52/rsync/io.c(453) [sender=2.6.9]

I am specifying my user name in my plist file and when debugging, the script says it's being run by that same user both when running it from the terminal and from launchd alike.

My BackupDaemon.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>EnvironmentVariables</key>
    <dict>
        <key>HOME</key>
        <string>/Volumes/Norman Data/me</string>
    </dict>
    <key>GroupName</key>
    <string>staff</string>
    <key>InitGroups</key>
    <true/>
    <key>Label</key>
    <string>BackupDaemon</string>
    <key>Program</key>
    <string>/Volumes/Norman Data/me/backup</string>
    <key>StandardErrorPath</key>
    <string>/tmp/BackupDaemon.err</string>
    <key>StandardOutPath</key>
    <string>/tmp/BackupDaemon.out</string>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>16</integer>
        <key>Minute</key>
        <integer>6</integer>
    </dict>
    <key>UserName</key>
    <string>me</string>
</dict>
</plist>

I can't figure out for the life of me what the issue might be or how to even go about further debugging. Any insight anyone has to share will be greatly appreciated. Thx!

 

[daveh0]

This is not the ideal fix, but changing this from a Global Daemon to a User Agent (meaning it only runs when I'm logged in and won't if another user account on this machine is logged in) by moving the .plist file to ~/Library/LaunchAgents cleared up the SSH issue.

But being a backup script, it *should* run regardless of which account is logged in. The documentation states that using the UserName key as I have should cause the script to run as the specified user when running as a Global Daemon (.plist file stored in /Library/LaunchDaemons/), but this is not happening here.

Can anyone think of any reason(s) why this may be or what I'm missing?

 

[chown33]

There might be something amiss in the execution context. The first things that come to mind are user ID, working directory, and environment variables. User limits is another possibility, but lower in likelihood, in my experience.

You can look up what 'ssh' needs from its execution context, in the way of env vars, dirs, etc.

You can troubleshoot the execution context by running a shell script in place of your backup program. Have it output the relevant context to a file. Then look at the file using a text editor, and see what it says.

A typical script (not tested) :

!#/bin/bash

umask 000

mkdir -p /Users/Shared/test/

OUT="/Users/Shared/test/log.txt"

id >"$OUT"

pwd >>"$OUT"

printenv | sort >>"$OUT"

ulimit >>"$OUT"

You should also look at the ownership and permissions on the /Library/LaunchDaemons/ plist file. As I recall, those will only run if owned by root, and only root has write permission. Allowing non-root-owned plist files in that location would be a security hole.

 

[daveh0]

ok, so here is the output of your script when run as a Global Daemon (plist file owned by root and located in /Library/LaunchDaemons/) as my user:

uid=501(me) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh)


/
HOME=/Volumes/Norman Data/me

LOGNAME=me

PATH=/usr/bin:/bin:/usr/sbin:/sbin

PWD=/

SHELL=/bin/bash

SHLVL=1

TMPDIR=/var/folders/pg/lrb76l2j4qd85qvhxf9cz5sc0000gn/T/

USER=me

XPC_FLAGS=0x0

XPC_SERVICE_NAME=0

_=/usr/bin/printenv

unlimited

and here it is when it run it from the terminal as my user:

uid=501(me) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh)


/Volumes/Norman Data/me/Scripts/ld_test

Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.1xONfU9bCR/Render

BLOCKSIZE=1k

EDITOR=/usr/bin/vim

HOME=/Volumes/Norman Data/me

LANG=en_US.UTF-8

LOGNAME=me

PATH=/usr/local/sbin:/usr/local/mysql/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin/:/Volumes/Norman Data/me/.composer/vendor/bin

PROJECT_HOME=/Volumes/Norman Data/me/Development

PWD=/Volumes/Norman Data/me/Scripts/ld_test

SHELL=/bin/bash

SHLVL=2

SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.NxAst9Rhqb/Listeners

TERM=xterm-256color

TERM_PROGRAM=Apple_Terminal

TERM_PROGRAM_VERSION=404.1

TERM_SESSION_ID=543B4C8A-3EDC-4EE7-84B7-8BF8520ADC1A

TMPDIR=/var/folders/pg/lrb76l2j4qd85qvhxf9cz5sc0000gn/T/

USER=me

VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3

WORKON_HOME=/Volumes/Norman Data/me/.virtualenvs

XPC_FLAGS=0x0

XPC_SERVICE_NAME=0

_=/usr/bin/printenv

unlimited

There are definitely some differences - I'm not really sure what I'm looking for. Does this tell you anything?

 

[chown33]

The most obvious difference directly related to 'ssh' is this env var, which is absent from the first output:

 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.NxAst9Rhqb/Listeners

The man page for 'ssh' (man ssh) says this:

     SSH_AUTH_SOCK         Identifies the path of a UNIX-domain socket used to
                           communicate with the agent.

Some of the other env vars may also be relevant. For example, PATH differs between the two.

The option '-l' passed to bash will make it act like a login shell, which means (among other things) it loads profile files from the home directory. Another possibly useful option: -c


Finally, the Code tag is better than the Quote tag for posting code.

 

[daveh0]

The most obvious difference directly related to 'ssh' is this env var, which is absent from the first output:

 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.NxAst9Rhqb/Listeners

The man page for 'ssh' (man ssh) says this:

     SSH_AUTH_SOCK         Identifies the path of a UNIX-domain socket used to
                           communicate with the agent.

Some of the other env vars may also be relevant. For example, PATH differs between the two.

The option '-l' passed to bash will make it act like a login shell, which means (among other things) it loads profile files from the home directory. Another possibly useful option: -c


Finally, the Code tag is better than the Quote tag for posting code.
View attachment 899434

Fixed QUOTE/CODE - my bad... not sure where my head was.

OK, so perhaps if I can get Bash to load my entire profile it will pick up the necessary ENV variables. And if not, I may have to pass them through the PLIST file as I am currently passing $HOME. Does that sound about right?

 

[Stephen.R]

Does your SSH key have a passphrase, and did you load the keys using the keychain agent (so the passphrases are are provided so long as your keychain is unlocked)?

 

[daveh0]

Does your SSH key have a passphrase, and did you load the keys using the keychain agent (so the passphrases are are provided so long as your keychain is unlocked)?

It does have a passphrase. I don't have to re-enter the password after a logout or reboot - does that mean it's using the keychain agent?

 

[daveh0]

The most obvious difference directly related to 'ssh' is this env var, which is absent from the first output:

 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.NxAst9Rhqb/Listeners

The man page for 'ssh' (man ssh) says this:

     SSH_AUTH_SOCK         Identifies the path of a UNIX-domain socket used to
                           communicate with the agent.

Some of the other env vars may also be relevant. For example, PATH differs between the two.

The option '-l' passed to bash will make it act like a login shell, which means (among other things) it loads profile files from the home directory. Another possibly useful option: -c


Finally, the Code tag is better than the Quote tag for posting code.
View attachment 899434

I imported SSH_AUTH_SOCK and it's showing up in the test output. i added an SCP command to the test file to see if it would connect using my user's key now that this other ENV variable was available to no avail - same permission denied error as documented in OP.

 

[Stephen.R]

It does have a passphrase. I don't have to re-enter the password after a logout or reboot - does that mean it's using the keychain agent?

Yep, probably.

If you want something to run unattended (ie not reliant on a logged in user) over SSH typically you'd use a key without a passphrase, because there's usually no way to provide it securely anyway.

Try generating adding another SSH key pair, without a passphrase this time, and then put the public key on the ssh server you want to backup to, and reference the private (passphrase-less) key in your backup command.

 

[daveh0]

Yep, probably.

If you want something to run unattended (ie not reliant on a logged in user) over SSH typically you'd use a key without a passphrase, because there's usually no way to provide it securely anyway.

Try generating adding another SSH key pair, without a passphrase this time, and then put the public key on the ssh server you want to backup to, and reference the private (passphrase-less) key in your backup command.

That worked!! Is there any risk of my password being stolen/exposed when using a passphrase-less key? I guess I mean, does standard/best practice dictate that I should create a separate account on the remote machine for this or am I ok using my personal account?

 

[Stephen.R]

That worked!! Is there any risk of my password being stolen/exposed when using a passphrase-less key? I guess I mean, does standard/best practice dictate that I should create a separate account on the remote machine for this or am I ok using my personal account?

That is a valid concern and it will depend entirely on the circumstances.

If you're just using rsync, and you have admin access (to install software) one the box, rrsync tied to the 'insecure' key may be an option: https://serverfault.com/a/965929

 

[daveh0]

Unfortunately i don't have that access on the remote box as it is shared hosting. i'm the only one allowed to touch the local machine. Am I asking for trouble with this setup?

 

[Stephen.R]

It's not an ideal scenario, but then if the remote machine is shared hosting, we're already way beyond the realm of 'ideal' when it comes to secured backups anyway.

 

[daveh0]

It's not an ideal scenario, but then if the remote machine is shared hosting, we're already way beyond the realm of 'ideal' when it comes to secured backups anyway.

Gotcha - my local NAS exploded the other day so I kind of freaked out and threw together this script just to have *something* in pace while i figure out whether I'm going to try to get some new hardware and keep everything local or just bite the bullet and spring for a cloud backup service. Good learning experience none the less. Thanks for the help/guidance.

 

[Stephen.R]

No worries, makes sense.