ES256 vs RS256 vs Ed25519 for Passkeys

Three COSE algorithms cover essentially every credential you will ever receive, and the choice between them is expressed as an ordered list in your registration options. This page covers what actually differs between them, how to order the list, and why every branch must be exercised by tests rather than only by production. It sits under Cryptographic Algorithms Supported by WebAuthn, which covers negotiation in general.

Algorithm Reference

Identifier Name Key material Signature Support
-7 ES256 P-256 point, compact DER-wrapped, ~70–72 bytes universal
-257 RS256 RSA modulus and exponent 256 bytes for a 2048-bit key common on some enterprise hardware
-8 EdDSA Ed25519 point, 32 bytes fixed 64 bytes real but not universal

Everything else in the COSE registry is either unsupported by authenticators in practice or unsupported by the verification stacks you are likely to use.

Three algorithms, three verification paths The differences that matter are in signature encoding, key size and hashing behaviour. What actually differs between the three ES256 (-7) P-256 with SHA-256; DER signature to convert; universally supported RS256 (-257) RSA with SHA-256; 256-byte signature; large keys EdDSA (-8) Ed25519; no separate hash parameter; fixed 64-byte signature anything else not worth offering — you will simply never receive it

Root Cause Analysis

1. Offering only ES256. It works for almost everyone, and it silently excludes users whose authenticator produces RSA keys. Those users cannot enrol at all, and the failure gives them nothing to act on.

2. Offering an exotic algorithm first. The authenticator takes the first entry it supports, so an unusual preference at the top routes a minority of devices down a code path that receives almost no production traffic and therefore almost no debugging.

3. Hard-coding the verification path. A verifier that assumes ES256 works until the first RSA credential arrives, at which point it fails with an error that looks cryptographic and is actually a dispatch bug.

4. Treating EdDSA as a drop-in. It takes no separate hash parameter, and libraries that require one will either throw or, worse, hash the input twice. Its signature length is fixed, which makes it easy to detect and easy to mis-handle.

Ordering the list you send The authenticator takes the first entry it can implement, so the order is a real policy decision. Order matters more than membership your list ordered by preference authenticator takes the first it supports response reports the choice storage key plus its algorithm Putting an uncommon algorithm first means the rare devices that support it use a code path you exercise far less than the common one.

Step-by-Step Resolution

Step 1 — offer ES256 first

It is supported everywhere, its keys and signatures are small, and it is the path your tests and your production traffic will exercise most.

Step 2 — include RS256

The cost is one extra verification branch and a larger stored key. The benefit is that users on RSA-producing authenticators can enrol at all.

Step 3 — include EdDSA if your stack verifies it cleanly

Check that your verification library accepts an Ed25519 key without a hash parameter before offering it. Offering an algorithm you cannot verify produces credentials that can never be used.

Step 4 — store the negotiated algorithm

The response reports which was chosen. Persist it in its own column and dispatch on it at verification time; never infer it from key length or from what you offered.

Step 5 — exercise every branch in tests

A branch that only runs in production is a branch that will first fail in production, on a user who has no idea why.

Where the verification paths diverge A single verify helper hides at least one assumption; the three algorithms differ in more than a parameter. Two shapes of the same operation ES256 and EdDSA compact keys and signatures ES256 needs DER to raw conversion EdDSA takes no hash parameter fast to verify RS256 256-byte signatures keys an order of magnitude larger PKCS#1 v1.5 padding slower, and larger in transit Sizing matters when an allow-list carries several RSA credentials over a constrained transport.

Verification and Testing

Capture one real registration and one real assertion per algorithm you accept, and assert that each verifies end to end. This is the only test that catches the combination of decode, dispatch and signature-format bugs, because each of them individually produces a plausible intermediate result.

Assert signature lengths explicitly in the fixtures. Sixty-four bytes means raw ECDSA or EdDSA; around seventy means DER-wrapped ECDSA; two hundred and fifty-six means RSA. A length assertion is the cheapest possible guard against a conversion being removed.

Add a negative test that runs an RSA credential through the elliptic-curve branch. It should fail at the dispatch, loudly, rather than at the signature check, quietly.

Pitfalls

1. Inferring the algorithm from the key. Store what the authenticator reported; do not re-derive it.

2. Sharing one verify helper. The three paths differ in hashing, padding and signature encoding.

3. Offering an algorithm your verifier cannot handle. The credential enrols and then can never be used.

Frequently Asked Questions

Is ES256 strong enough?

Yes. P-256 with SHA-256 is a standard, widely reviewed choice, and it is what essentially the whole ecosystem uses. The reasons to offer alternatives are compatibility and, occasionally, an organisational requirement — not a weakness in ES256.

Why would an authenticator produce an RSA key?

Some platform and enterprise authenticators are built on stacks where RSA is the native or the only available key type. It is not a sign of an old or insecure device; it is a property of the underlying key store.

Should I remove RS256 once nobody is using it?

Only after checking that nobody actually is. Removing it does not affect existing credentials, which continue to verify with the stored algorithm — it only stops new enrolments from choosing it. That said, the cost of keeping the branch is small, and the cost of a user who cannot enrol is not.

Does the algorithm affect how the credential behaves?

No. Discoverability, backup state, user verification and every other property are independent of the signature algorithm. It affects only the verification path and the storage size.

Can one account hold credentials with different algorithms?

Yes, and it is common in a mixed device population. Each credential carries its own algorithm, which is exactly why the value must be stored per credential rather than configured globally.

How much larger is an RSA key in practice?

Roughly an order of magnitude compared with a P-256 key. It matters for column sizing, and it matters when an allow-list carrying several credentials is sent to an authenticator over a constrained transport where message size is limited.

What happens if I send an empty algorithm list?

The request is malformed and the ceremony fails. An empty list does not mean “anything is acceptable”; it means you have specified no acceptable algorithm, which no authenticator can satisfy.

Does the choice of algorithm affect performance at scale?

Marginally, and almost entirely on the RSA path. Verifying an RSA signature costs more than verifying an elliptic-curve one, and RSA keys and signatures are larger both in storage and in transit. For a service handling ordinary sign-in volumes none of this is a constraint worth optimising for — the database lookups around the verification cost more than the verification itself. It becomes worth knowing in two places: when sizing columns for a credential table that will hold millions of rows, and when an allow-list carrying several RSA credentials approaches a message-size limit on a constrained transport such as NFC, where a large request can simply fail to reach the authenticator.

Should the algorithm ever be surfaced to a user?

No. It is an implementation detail with no bearing on anything a person can decide, and naming it in an interface invites a question nobody in your support queue can usefully answer. Where a difference genuinely reaches the user — an authenticator that cannot enrol because no offered algorithm is supported — the message should describe the device situation rather than the negotiation, and the fix belongs in your options list rather than in the user’s hands.

One further note for anyone auditing an existing service: the presence of a stored algorithm column is not proof that the verification path reads it. Search for the point where the verifier is constructed and confirm the value flows in from the credential row rather than from a constant, because a column that is written and never read looks identical to one that is doing its job.

Related