X-amine509: Predicting the Practical Risk Level of Enterprise X.509 Certificates
Digital certificates establish trust between services and devices, and weak keys or misconfigured certificates can let attackers impersonate trusted systems, intercept sensitive communications, or gain unauthorized access. Deterministic analysis tools that precisely identify standards violations are indispensable for remediation, but applying them exhaustively across millions of certificates is operationally impractical. In our experiments, running the full suite of deterministic checks on roughly one million certificates took over 24 hours.
X-amine509 is a two-stage triage system. A machine learning model rapidly ranks certificates by predicted risk, and only the highest-risk items are routed to full deterministic analysis. Risk is a composite score derived from 177 defect checks grounded in CA/Browser Forum Baseline Requirements, NIST IR 8547/SP 800-57, and cryptographic strength criteria, weighted across four severity tiers ranging from cryptographic breaks to minor compliance deviations. We collected and scored 1,027,714 certificates from Fortune 500, .gov, and .edu domains, then compared 25 models.
On a held-out test set of 201,976 certificates, Extra Trees achieved the best fit (R² of 0.993, MAE of 2.26), while a Decision Tree reached R² of 0.986 at about 3.7 million predictions per second on a single machine. Severity-tier classification reached 99.76% accuracy with 98.90% recall on critical-tier certificates, and aggregate NDCG exceeded 0.997.
To test durability, we collected another 571,374 certificates thirteen months later. Without retraining, the Extra Trees and Decision Tree models kept R² of at least 0.915, severity-tier accuracy of at least 99.52%, and critical-tier recall of at least 97.03%, with the Decision Tree retaining 98.06%. Random Forest had lower average error on the newer certificates but missed more critical-tier certificates than the Decision Tree, so we recommend the Decision Tree for pairing speed with the critical-tier recall a security workflow actually needs.
Feature importance analysis identifies validity period, Extended Key Usage configuration, negative serial number encoding, and self-signed status as the strongest risk predictors. The recommended deployment uses ML to prioritize detailed analysis while retaining targeted deterministic checks for the most consequential defects regardless of the predicted score.
Co-authored with Shubh Patel, JD Kilgallin, and Caleb Shorter during an AI and ML internship at Keyfactor.