Assuming you are running on a modern Linux or similar system, the encryption part should be fairly simple.
(Edit: Reading your question again, I see you started by asking about authentication and then asked the specific question about RSA encryption which I've answered here. You probably want to set things up as I've described here, then go and investigate pam_pkcs11 which is designed to do this kind of thing.)
Setup
Make sure your Yubikey has enabled 'PIV' mode, and you've installed the OpenSC PKCS#11 module which can talk to it. Also install the OpenSSL PKCS#11 'engine'.
Use the yubico-piv-tool or any other method you like to install an RSA key into the slot of your choice, let's say the 'Card Authentication slot 9e for example. You can either import an existing key, or create one that's never existed outside the Yubikey.
Signing
Now it's simple. Any software that can use an RSA key from a file ought to accept a standard PKCS#11 URI identifying the key in the Yubikey instead.
The URI for the key in the Card Authentication slot will be something like 'pkcs11:manufacturer=piv_II;id=%04'.
So you can do something like
echo "test payload" | openssl dgst -sha256 -sign 'pkcs11:manufacturer=piv_II;id=%04' -hex
... except OpenSSL doesn't quite get this right yet, so you have to add
-engine pkcs11 -keyform engine
to the above command line because all crypto software authors hate their users and it doesn't bother to infer those obvious things from the fact that you gave it a PKCS#11 URI. This is also slightly out of date because newer OpenSSL uses "providers" instead of engines, and I think it still doesn't just get things right for itself so you may need a slightly different pointless arcane incantation to make it do so.
Decent software won't need those extra hints, and will just take the URI in place of a filename and do the right thing. File bugs if not.
Example
$ yubico-piv-tool -s9e -ARSA2048 -agenerate | tee pubkey.pem -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr9LrzjNbRABqhDQrGi3l VcQhmUu0lls8k4XnO8c/U0oS6IvH4H7HuqXFfwThYofxIgA2eIXuRXf+V/CSWtXN 40Bb10QKcTXVATm05+KFNWg1GCVg2yrvsUOQSd6MOxAa5goUUi2xOjeLFZRvXuvt YmLytwY77YqE0WOHYfYuk9kolueZHhq4BSOVRmQpZxKd6/MkWlT46SPc3Bwbyx41 t2U42vlnHYuma3NF6qI+a+LaMyvkFVBkRM6A1WB2u5jjl/ZQmyYsuqg2e8xu7P8m 5/GUH9HjD074+ea1NEdMncPhKjO+pL24BSebtIPUmzEJIh6kCaweJYiMKYuLx15H HQIDAQAB -----END PUBLIC KEY----- Successfully generated a new private key. $ echo Test | openssl pkeyutl -encrypt -pubin -inkey pubkey.pem > Test.enc $ openssl pkeyutl -decrypt -in Test.enc -engine pkcs11 -keyform engine -inkey 'pkcs11:manufacturer=piv_II;id=%04' Engine "pkcs11" set. Test
You can also do signatures...
$ echo Test | openssl dgst -sha256 -sign 'pkcs11:manufacturer=piv_II;id=%04' -engine pkcs11 -keyform engine > signature.bin $ echo Test | openssl dgst -sha256 -verify pubkey.pem -signature signature.bin Verified OK
TL;DR
The different slots in the Yubikey PIV have different policies, especially around whether the PIN is required. Read the documentation I linked. See https://www.infradead.org/openconnect/pkcs11.html for more hints on finding the URI for a given object in a token.