JWT Authentication in Microservices: What I Learned the Hard Way

Three years ago I deployed a JWT-based auth system across six microservices. The following Monday, at 2:47 PM, the pager went off. Every single service was rejecting valid tokens. Users were locked out. The CTO was standing behind my desk.

The cause? I had deployed a config change that rotated the JWT signing key but the services cached the old key and the rotation logic had a race condition. Six hours of debugging later, I had a new appreciation for how fragile JWT authentication can be in distributed systems.

This article covers what I learned from that incident and from hundreds of code reviews since.

The Default That Will Bite You: Shared Secrets

Most JWT tutorials show this:

// jwt.io style example — fine for a monolith, dangerous for microservices
const jwt = require("jsonwebtoken");
const token = jwt.sign({ userId: 123 }, "mySecretKey123");
const decoded = jwt.verify(token, "mySecretKey123");

On Stack Overflow, the question "JWT authentication for microservices — share secret or use public key?" has over 120,000 views. The top-voted answer (900+ upvotes) is clear: use asymmetric keys.

The problem with a shared secret in microservices is twofold:

  1. Every service can sign tokens. If the user-service is compromised, the attacker can mint valid tokens for any user. With RS256 (asymmetric), only the auth-service holds the private key. Other services only have the public key—they can verify, but they cannot forge.

  2. Key rotation requires synchronized deployment. You cannot rotate the secret gradually. Every service must switch at exactly the same moment, which is nearly impossible in a distributed deployment.

// Correct approach: RS256 with public/private keys
const jwt = require("jsonwebtoken");
const fs = require("fs");

// Auth service (signs tokens)
const privateKey = fs.readFileSync("private.pem");
const token = jwt.sign(
  { sub: user.id, roles: ["admin"] },
  privateKey,
  { algorithm: "RS256", expiresIn: "15m" }
);

// Other services (verify tokens)
const publicKey = fs.readFileSync("public.pem");
const decoded = jwt.verify(token, publicKey, {
  algorithms: ["RS256"]  // Never omit this!
});

Notice the algorithms: ["RS256"] option. Without it, an attacker can change the alg header in the token to HS256 and sign with the public key (which is public). This is called the algorithm confusion attack, and a Stack Overflow thread about it has over 55,000 views.

Token Revocation: The Elephant in the Room

JWTs are stateless by design. The server does not store session data. This is great for scalability but terrible for revocation.

Here is the scenario that plays out in every real deployment: a user changes their password, or an admin deactivates an account. The user's existing JWT is still valid for another 14 minutes (or however long your expiry is).

A post on r/webdev asking about JWT revocation garnered 280+ comments. The consensus was not pretty: JWT revocation is an unsolved problem that everyone solves differently.

After trying three approaches, here is what I settled on:

// Short-lived access tokens + long-lived refresh tokens
// Access token: 15 minutes (stored in memory on client)
// Refresh token: 7 days (stored in HttpOnly cookie, hashed in DB)

// On password change or admin deactivation:
await db.collection("refresh_tokens").deleteMany({ userId });

// The access token expires in 15 minutes naturally.
// After that, the client tries to refresh and gets a 401.
// The client redirects to login.

The key insight: shortened expiry is your revocation mechanism. If tokens expire every 15 minutes, the maximum damage window is 15 minutes. For most applications, this is acceptable.

If you need instant revocation for compliance reasons (finance, healthcare), maintain a Redis denylist:

// On revocation event
await redis.set(`revoked:${tokenJti}`, "true", "EX", 900); // 15 min TTL matches token expiry

// In verification middleware
app.use((req, res, next) => {
  const token = extractToken(req);
  const decoded = jwt.verify(token, publicKey);

  // Check denylist
  if (await redis.get(`revoked:${decoded.jti}`)) {
    return res.status(401).json({ error: "Token revoked" });
  }
  next();
});

This adds a Redis lookup per request but keeps the 15-minute revocation window down to real-time.

The Clock Skew Nightmare

Here is a bug that took me two days to find: one of my microservices was running on a server with its clock 47 seconds behind the auth service. The JWT had exp: Math.floor(Date.now() / 1000) + 900 in the future when signed, but "past" when verified by the slow-clock server.

On Stack Overflow, this exact problem has over 35,000 views and the top answer recommends the clockTolerance option:

const decoded = jwt.verify(token, publicKey, {
  algorithms: ["RS256"],
  clockTolerance: 30  // Allow 30 seconds of clock skew
});

I now set clockTolerance: 30 on all verification calls. But the real fix was adding NTP monitoring to our infrastructure alerts.

Service-to-Service Authentication

When microservice A needs to call microservice B, you have two options: pass the user's token through, or use a separate service account token.

Passing the user token through (token exchange) preserves the user context end-to-end:

// Service A receives the user token, forwards it to Service B
const userToken = req.headers.authorization; // "Bearer eyJhbGci..."
const response = await fetch("http://service-b/internal/data", {
  headers: { Authorization: userToken }
});

But this breaks when Service A needs to do work asynchronously or when the user's token expires before the workflow completes. In those cases, use a service account JWT:

// Service A signs its own JWT with a service identity
const serviceToken = jwt.sign(
  {
    sub: "service-a",
    aud: "service-b",
    type: "service"
  },
  servicePrivateKey,
  { algorithm: "RS256", expiresIn: "5m" }
);

A discussion on r/programming about this pattern has over 400 comments debating whether service-to-service calls should carry user context at all. The majority view: pass user context for synchronous requests, use service identity for async or batch operations.

What I Put in JWTs (And What I Do Not)

After several mistakes, here is my current claim set:

{
  "sub": "user_8a7b3c2d",
  "aud": "api.devformatters.com",
  "exp": 1893456000,
  "iat": 1893455100,
  "jti": "unique-token-id-abc123",
  "roles": ["admin"],
  "tenant_id": "t_42"
}

What I explicitly do not put in JWTs:

  • Full user profile data (name, email, avatar). The token is signed but the payload is readable by anyone. Base64 decoding the payload reveals everything.
  • Permissions list. Permissions change frequently. The token outlives the permission update. Store a role and look up permissions on the server.
  • Sensitive data (passwords, API keys, PII). The payload is not encrypted unless using JWE (JSON Web Encryption), which is rare in practice.

Related Searches

  • jwt microservices best practices
  • jwt token revocation strategy
  • jwt asymmetric vs symmetric signing
  • jwt clock skew tolerance
  • service to service authentication jwt
  • jwt refresh token rotation
  • jwt algorithm confusion prevention
  • rs256 vs hs256 jwt
  • jwt claims best practices
  • microservices authentication patterns

Frequently Asked Questions

Should microservices share a JWT secret?

No. Use asymmetric signing (RS256 or ES256). Each service only needs the public key to verify tokens. Only the auth service holds the private key. This limits the blast radius if any individual service is compromised.

How do you handle token revocation in microservices?

Three strategies: (1) short-lived tokens (15 min) so revocation happens naturally, (2) refresh token rotation with a database-backed denylist, (3) Redis denylist for instant revocation of specific token IDs. Combine (1) with either (2) or (3) depending on your requirements.

What JWT claims should I include?

Standard: sub (user ID), aud (audience), exp (expiration), iat (issued at), jti (unique token ID). Custom: roles (not individual permissions), tenant_id if multi-tenant. Never include sensitive data like email or real name.

How do I handle JWT in service-to-service calls?

For synchronous calls with user context, forward the user's token. For async calls or batch processing, use a separate service account JWT signed with the service's own key. Set a short expiry (5 minutes) for service tokens.

What happens during JWT key rotation?

Publish the new public key alongside the old one. Services should support both during the rotation window. Set a kid (key ID) header in the JWT so verifiers can select the correct key. Rotate gradually, not all at once.

Why did my JWT verification fail after deployment?

Most common cause: clock skew between services. Set clockTolerance: 30 seconds on verification. Second most common: the public key path is wrong in the deployment environment. Verify that fs.readFileSync("public.pem") resolves to the correct file.

Final Thoughts

JWT authentication in microservices is not as straightforward as the introductory tutorials suggest. The challenges come from distribution: clock skew, key distribution, token revocation, and service-to-service auth.

The pattern I have settled on after three years: RS256 with 15-minute access tokens, 7-day rotating refresh tokens, Redis denylist for instant revocation, and clockTolerance: 30 on every verification call. It is not perfect, but it has survived two production incidents that would have taken down our previous session-based system.

Use the JWT Decoder tool on DevFormatters to inspect your own tokens and verify that the claims are structured correctly before deploying to production.