I mentioned the other day how sad it is that so many links end in 404 after a short while, and I got a reply that it might be 410. Actually, it’s also often a 301.
Be that as it may, and to distance myself as much as possible from that one particular person’s site, I have replaced all links to what was once called twitter to this page.
It’s very sad, as it breaks quotes in articles like this one, but so be it.
Directing an SSH agent socket (UNIX domain socket) into a podman container, is as easy as 1-2-mount, so I erroneously assumed that doing similarly for GPG would be as easy. It wasn’t, at least for me.
I rose at 06:00 and yelled Eureka! at 15:05 local time. It took me so long because I tried to follow GNU documentation which speaks of using extra-socket. To cut a far too long story short, on the Debian host I’m doing this none of that business worked: the gpg-agent was just not creating the UNIX domain socket where I configured it to.
What I then did was to completely deactivate system’d socket activation for GPG: stop and disable the service and the socket activations. The main reason: changes in the unit file for the service, e.g. --extra-socket, weren’t being applied.
Then I launched the daemon manually and began looking at some logs. (No, I’m not a guru, but I hoped I’d see lots of things; omg did I.)
$ gpg-agent --daemon --debug-level guru > /tmp/glog 2>&1
Slowly but surely, after downloading the source to GPG and looking at what the agent does, it dawns on me that something magic is happening, and I blame either the distro and/or systemd. Be that as it may, the debian files in /usr/share/doc don’t mention anything about deactivating the extra socket, so shrug.
What I can’t get to work at all are the settings in ~/.gnupg/gpg-agent.conf, so I ignore that.
Anyway, I’m a bit further, but not really much, but it’s clear that it’s not gpg-agent’s fault.
Enough rambling, as I can’t remember all I’ve done, let me jot down how I solved the puzzle:
I reactivated all the systemd stuff, and everything’s back to normal. gpg-agent is creating the files in /run/user/1003/gnupg/ and when I try to kick the agent by decrypting a file on the host, the following files are created
$ gpgconf --kill gpg-agent
$ rm -r /run/user/1003/gnupg/
$ gpg --batch --use-agent --decrypt project/vault_passphrase.gpg
$ ls -l /run/user/1003/gnupg/
total 0
srwx------ 1 jpm staff 0 Apr 16 12:58 S.gpg-agent
srwx------ 1 jpm staff 0 Apr 16 12:58 S.gpg-agent.browser
srwx------ 1 jpm staff 0 Apr 16 12:58 S.gpg-agent.extra
srwx------ 1 jpm staff 0 Apr 16 12:58 S.gpg-agent.ssh
So be it. That’s where the sockets are. :shrug: (Note, I’m displeased, but :shrug:)
Now back to podman. The intention is to carry a socket across, so that the container can access my agent. This is the magic incantation:
--container-volume-mount /run/user/1003/gnupg/S.gpg-agent:/root/.gnupg/S.gpg-agent:Z
When the container launches (target user is root), my agent socket (user 1003) is mounted at /root/.gnupg in the target, and the root user there has access to it.
This actually works, and I want this to be able to provide encrypted vault secrets.
Total overkill, and half a day’s worth of smoke arising from the top of my skull, but if somebody asks “can it?”, I can answer: “yes, it can!”
The actual fun begins then, with Ansible accessing the vault, configured in an ansible.cfg for the project:
[defaults]
vault_password_file = vault_pass.sh
The vault_pass.sh program which is taken across in the ansible-runner project now does this (output to stderr so that it’s carried back to the host which invoked ansible-runner):
gpg --list-keys >&2 # will create the .gnupg directory
cat > ~/.gnupg/gpg.conf <<EOF1
use-agent
EOF1
cat << EOF2 | gpg --import -
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGNBGYc9KgBDACm0FOP70HFdSfGVL9W8X0HkasdTxYrz5RWJmM+Z3oxBZEpzAuP
4572CAdZC4MS/8B/ABlnojJBegrATySNu+fB7wHURKkz4BOEnF3oobJQoj+PYOT4
...
rzkFQg==
=xTtx
-----END PGP PUBLIC KEY BLOCK-----
EOF2
gpg --homedir /root/.gnupg --batch --use-agent --decrypt vault_passphrase.gpg
echo "Outta here" >&2
Output (stdout) from gpg is consumed by Ansible to unlock vault secrets.
And yes, this does function:
$ ansible-runner run \
--container-volume-mount /run/user/1003/gnupg/S.gpg-agent:/root/.gnupg/S.gpg-agent:Z \
. -p v1.yml
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 353C430586B83FE5: public key "JP Ansible (Runner Vault tests) <jp@example>" import
ed
gpg: WARNING: server 'gpg-agent' is older than us (2.2.40 < 2.3.3)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
gpg: Total number processed: 1
gpg: imported: 1
gpg: encrypted with rsa3072 key, ID 3F9FC1C9EA3FE42F, created 2024-04-15
"JP Ansible (Runner Vault tests) <jp@example>"
gpg: WARNING: server 'gpg-agent' is older than us (2.2.40 < 2.3.3)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
gpg: problem with fast path key listing: IPC parameter error - ignored
Outta here
PLAY [Vault test] **************************************************************
TASK [ping] ********************************************************************
ok: [w00]
TASK [Can we unvault this var?] ************************************************
ok: [w00] => {
"msg": "The cow jumped over the moon."
}
The variable was unvaulted.
One important bit: the socket activation of gpg-agent only occasionally triggers the pinentry program on the host; I’ve noticed it works smoothest if I do so myself by decrypting a file on the host. (I’m not now feeling up to how to configure TTLs on the agent-cached secrets; I hope that’s more straightforward than what I dug into today.)
Moo.
In desperation I thought I’d see what happens when gpg-agent finds a file in the place it wants to create a directory. I wish I hadn’t done that, but see for yourselves. The result is what I’ve spent 3/4 of a day trying to accomplish.
Yeah, my use of man gpg-agent at the end is sarcastic. I could have written worse: info gnupg
I download the Debian source package for gpg-agent, and therein I do a grep -ri run.user .. To be fair, they’ve documented it somewhat:
Users who don’t want systemd to manage their gpg-agent in this way for all future sessions should do:
systemctl --user mask --now gpg-agent.service gpg-agent.socket gpg-agent-ssh.socket gpg-agent-extra.socket gpg-agent-browser.socket
The reason I didn’t find that sentence yesterday? I looked in /usr/share/doc/gnupg. Beginner error. It’s in /usr/share/doc/gpg-agent (list of files). Sighs.
But there’s no word on the fallback to $HOME/.gnupg that I can find.
My preferred Let’s Encrypt (a.k.a. ACME) challenge is the DNS-01 challenge. I’ve used a few different implementations, but have settled for the Golang executable with the cute name: lego.
DNS-01 enables me to have certificates (also wildcard certificates, i.e. *.example) issued from a machine which doesn’t require incoming HTTP traffic from an ACME server to perform verification. lego adds the challenge to the DNS via, say, RFC2136 Dynamic DNS updates, and the Let’s Encrypt servers check whether the challenge is at a particular name; if so, the certificate is issued, and lego places it on the file system.
I might want the certificates I have issued with DNS-01 deployed to multiple distinct machines for different purposes, specifically for wildcarded names. Note that I am discussing distributing the public certificates, not their keys.
With lego, run via cron, I configure a hook script which is run whenever the certificate is actually renewed:
$ lego \
--email jp3@mens.de \
--dns rfc2136 \
--domains 'www.example' \
renew \
--reuse-key \
--renew-hook="./hook-renew.sh"
The specified hook-renew.sh obtains the actual certificate and can further process it. For instance, I send myself a reminder that the certificate has been issued, and upload it to a repository. (To be honest, the email also contains instructions to myself what to do in case any of the electric bits don’t work, i.e. how to do a manual deployment.)
# hook-renew.sh
compose-message.py "$LEGO_CERT_PATH" | /usr/local/bin/msmtp jpm
/usr/local/bin/ssh-agent ./gp.sh
The target repository has an SSH deploy key I add specifically for this purpose, meaning this SSH key has no passphrase, but it can write only into this particular repo. The gp.sh script is invoked by a “new” ssh-agent, so the ssh-add below is for that temporary agent:
# gp.sh (C)2023 by Jan-Piet Mens <jp@mens.de>
# push a public X.509 certificate
# add a password-less SSH key to the SSH agent which invoked us and call git
# push.
ssh-add /path/to/acme-pusher-key
# use openssl to produce the commit message which is read from stdin:
#
# serial=0429F8A41945B84985C704F44570636B13A8
# subject= /CN=www.example
# notBefore=Sep 11 05:40:15 2023 GMT
# notAfter=Dec 10 05:40:14 2023 GMT
#
openssl x509 -in .lego/certificates/www.example.crt -noout -serial -subject -dates |
git commit -F - -a
git push "$@"
On the receiving ends, i.e. on the systems which actually need the certificate, I have a cron job which periodically pulls the repository and checks whether a new certificate is available. These systems also have a dedicated passphrase-less SSH key which I configure as readonly in the repository. Here again, the script is invoked via ssh-agent:
ssh-add acme-reader-key
(cd repo && git pull --quiet)
# determine serial numbers of both certificates (installed and new)
# p := path, s := serial
p1="repo/.lego/certificates/www.example.crt"
p2="$target/tls.crt"
## update: valid point by Jeroen: it's better to test whether ## the files
## differ which would also catch changes in the intermediary issuers. And
## it's 'cheaper' to do a cmp(1) or compare the files' hashes.
s1=$(openssl x509 -serial -noout -in $p1)
s2=$(openssl x509 -serial -noout -in $p2)
if [ "$s1" != "$s2" ]; then
cp $p1 $p2
# the new certificate is installed; do whatever we have to,
# e.g. reload services, etc.
# ...
fi
The certificates are public, so I wouldn’t actually need an SSH deploy key to fetch them, but I do this on private repositories, so I do. Where they public, an unauthenticated git pull from an HTTPS URL would suffice.
Setup is a bit convoluted with what having to create SSH deploy keys for the repository and getting the bits and pieces right, but I’ve been using this for over half a year with 60-day certificate renewals without issues.
Jeroen points out intermediary issuers can change and suggests comparing the files; I’ve added comments to that effect in above script.
I awoke before dawn, and with still sleepy eyes tooted this, which I thought quite funny; it tickles my at times slightly warped sense of humor. Warning: some understanding of DNS and its utilities is required to get this.
I was very amused by some of the reactions I got to the post:
Thank you to all those who responded!
Click on the image above to get to the original thread.
I thought a rather complex Ansible template using quite a few filters was running particularly slowly, and decided to profile the playbook.
[defaults]
callbacks_enabled=ansible.posix.profile_tasks
The play templates out an OwnTracks configuration file, looping over roughly thirty users, with the loop taking 20 seconds.
- name: "template user .otrc files"
template:
src: otrc.j2
dest: "/tmp/a-{{ pathname }}"
loop: "{{ friends }}"Now, I’m aware Ansible isn’t the fastest beast on the planet, but this seemed a bit much. The play runs on and against localhost. I know that the module does its work in an idempotent manner, etc. with quite an overhead, but nevertheless I was curious whether I could speed it up.
system: template user .otrc files ------------------------------------- 19.89s
I replaced template by copy thereby removing Jinja functionality to see if my Jinja template itself was at fault, but there was no difference. (Template actually uses copy functionality for actually copying the templated result to the destination file.)
Experimentally I use the template filter to process Jinja into a string which is passed to a small filter I wrote called store(). This filter simply creates a file and drops the data in it.
- name: "template user .otrc files (jp: store)"
set_fact:
x: "{{ lookup('template', 'otrc.j2') | store(pathname) }}"
loop: "{{ friends }}"The result is quite dramatic for the same number of users in the friends array:
system: template user .otrc files (jp: store) -------------------------- 0.90s
The title of this post contains the words unfair comparison, and what I do here is unfair. The second variant isn’t at all idempotent but just clobbers the target file. Most importantly, this works for me only because I’m running it on localhost – filters are expanded in the templating system which runs on the controlling system. For deployment onto a remote node I’d have to employ copy whereupon I’d be back at square one.
Even so I find the results entertaining, and it demonstrates to me that the template filter can be quite useful.
To be clear: I will use the template module as in the first example above: whether this play takes twenty seconds more or less doesn’t matter in the slightest; users will typically run it only once in a blue moon.