Transparent by design
How AgentPulse scores reports
Transparent deterministic methodology, uncertainty, source weighting and vendor independence.
What we measure
AgentPulse measures community-reported coding-agent experience. Reports are subjective, selected from public sources, and are not controlled coding benchmarks. No score proves objectively better code.
AI extracts; application code scores
The classifier extracts relevant subjects, dimensions, sentiment and themes. Tool, provider, model and subscription plan remain separate. Model-only complaints do not automatically reduce tool scores. Announcements and vendor marketing carry zero community sentiment weight.
Weighting and shrinkage
Observation weight combines source, experience, classification and attribution confidence, recency, capped engagement, duplication and author frequency. GitHub defaults to 1.0 and Hacker News to 0.8. Recency halves after 12 hours, 3.5 days or 14 days for the corresponding windows. Dimension values range from -2 to +2; scores map to 0–100 and shrink toward a neutral prior of 50 with weight 10. Missing dimensions are omitted and profile weights redistribute proportionally.
Evidence and confidence
We report both raw sample size and Kish effective sample size. Independent duplicate groups and author-day groups cap evidence used for eligibility and confidence. A default leaderboard needs at least 20 effective independent reports and multiple community sources. Low samples cannot receive high confidence. Rankings remain withheld until classifier verification.
Trends
Trends compare equal adjacent periods. A change must exceed 3 points and 1.5 times combined approximate standard error, with at least 20 effective independent reports in both periods, before it is labeled significant. This is a heuristic uncertainty rule, not a claimed 95% confidence interval.
Limitations and independence
Source communities are not representative of all developers. Selection, language, attribution and classification errors can affect results. Prices come only from verified official sources. Sponsorships, affiliates and advertising never enter scoring. AgentPulse is independent of the analyzed vendors.