SAML SSO breaks client_credentials OAuth flow

Enforced SAML SSO on the org. The internal service account still needs client_credentials access for background jobs. Getting invalid_grant on /oauth/token now. The token endpoint doesn’t seem to accept the client secret anymore since the user session requires SAML. Is there a way to whitelist the OAuth client for programmatic access while keeping SAML enforced for humans? Here’s the error response.