The concept of ratings has been the accepted standard for making investment decisions. The first commercial credit reporting agency, the Mercantile Agency, was founded in 1841. While this relied on largely subjective methods of evaluation, it wasn’t until the 1960s, when credit reporting became computerized, that the industry consolidated and took off. Since then, credit and financial ratings models have progressed to become objective and trustworthy data points that inform lending decisions.
Similarly, SecurityScorecard founded security ratings to provide vendor risk management and security leaders with an objective outside-in view of the security of any company in the world. Just like the financial industry has a standardized rating system, organizations need a standardized measure of cyber health, with meaningful metrics and KPIs. As part of providing trustworthy ratings, understanding the impact of a change in a security rating is key and this is where transition matrices come in.
What is a transition matrix?
A transition matrix is a useful means of representing the impact of change in a system. It’s a compact way of showing the extent of change of state, for example, how many domains move from one categorization to another. Transition matrices are used in multiple industries, such as to show the result of medical treatment or ratings changes, and likewise also represent the extent of changes in our cybersecurity ratings.
Transition matrix in US credit ratings regulation
Here’s an example from the US SEC’s regulation of credit ratings agencies. The chart shows how ratings for companies have changed for organizations over the course of ten years — it’s a measure of the stability of ratings. Ideally, if you rate an organization with an A today, in a year’s time, it should still be an A. If you have lots of changes in your ratings, then your ratings are not reliable predictors of company performance.
Source: “Nationally Recognized Statistical Rating Organizations,” Securities and Exchange Commission, September 15, 2014
The horizontal axis of this table shows the ratings in 2000 and the vertical axis shows the ratings for the same companies in 2010. If you look at the row for ‘BB’ in 2000,1,000 companies were rated ‘BB’. Of these 1,000, ten years later, 2% were rated ‘A’, 4% were rated ‘BBB’, and 20% were rated ‘BB’, and so on. This shows you how ratings change over a decade. The ideal state is no changes, which is the diagonal shaded data in the table above. The figure below expands this description.
Source: “APPLICATION FOR REGISTRATION AS ANATIONALLY RECOGNIZED STATISTICAL RATING ORGANIZATION (NRSRO)”
Transition matrices can also be applied to the medical field. Imagine that there is a disease and two tests. How can you compare them? Let’s imagine there are 1,000 people in a study and they’re given both tests. For Test A, 200 people were positive and 800 were negative. For test B, 204 people were positive and 796 were negative. Here’s the partially completed transition table.
On the face of it, the two tests performed about the same, about 20% of people were positive. But did they detect the same people?
In the table below, cell +A+B represents patients who tested positive for test A and B. Cell -A-B is patients who were negative on both tests. Cell -A+B is patients who were negative on A and positive on B, and so on.
Now let’s fill in the table with data.
This table tells us something very important. The two tests are identifying different groups of people as positive. If the tests identified the same people, there would be no one in the +A-B and the -A+B cells. We know the tests are identifying different people, so we can make an informed decision about combining the two tests.
How to build a transition table in practice
Now let’s talk about building a transition table for cybersecurity. Imagine two algorithms that detect whether a domain belongs to a hosting company or not (Hosting and Regular). The old algorithm detects about 20% of domains as hosting and the new algorithm detects about 24% of domains as hosting. We think the new algorithm is more accurate than the old. On the face of it, the new algorithm is great. But is it? We can use the transition matrix to find out.
Here’s the old algorithm in table OLD:
Here’s the new algorithm in table NEW:
Let’s start building the matrix:
Let’s get the numbers. There are multiple ways to solve this problem, here’s just one.
OHT is old algorithm hosting, so it’s OLD.filter(F.col(‘Type’) == ‘Hosting’).count()
OHR is old algorithm regular, so it’s OLD.filter(F.col(‘Type’) == ‘Regular’).count()
NHT is new algorithm hosting, so it’s NEW.filter(F.col(‘Type’) == ‘Hosting’).count()
NHR is new algorithm regular, so it’s NEW.filter(F.col(‘Type’) == ‘Regular’).count()
NHOH is new algorithm hosting, old algorithm hosting, so it’s:
NEW.filter(F.col(‘Type’) == ‘Hosting’).merge(NEW.filter(F.col(‘Type’) == ‘Hosting’),
on=”Domain”, how=”inner”).count()
NHOR is new algorithm hosting, old algorithm regular, so it’s:
NEW.filter(F.col(‘Type’) == ‘Hosting’).merge(OLD.filter(F.col(‘Type’) == ‘Regular’),
on=”Domain”, how=”inner”).count()
NROH is new algorithm regular, old algorithm hosting, so it’s:
NEW.filter(F.col(‘Type’) ==’Regular’).merge(NEW.filter(F.col(‘Type’) == ‘Hosting’),
on=”Domain”, how=”inner”).count()
NROR is new algorithm regular, old algorithm regular, so it’s:
NEW.filter(F.col(‘Type’) == ‘Regular’).merge(OLD.filter(F.col(‘Type’) == ‘Regular’),
on=”Domain”, how=”inner”).count()
Let’s say you saw a matrix like the one below, what would you make of it?
Nine percent of your domains changed from Regular to Hosting with your new model. That sounds OK on the face of it. But five percent of domains went from Hosting with the old algorithm to Regular with the new. Is that acceptable? Possibly, and by using this matrix you can gain insight into important transitions you might otherwise miss.
SecurityScorecard’s transition matrix for the security industry
SecurityScorecard, like other credit rating agencies; completed this exercise for our security ratings. Our transition matrix shows that transition probability between one score and another remains relatively stable. The highest probability of transition between each state is to remain at the current score they exhibit.
How to read the matrix
A ratings transition matrix tells us on the X axis “If I have a score of X today”, X being our grade system (A,B,C,D,F); what is the probability that your score remains or “transitions” to another grade one year from today (the Y axis). Each box of the matrix represents the probability that the company’s rating either changes or remains the same depending on if you are looking at cross score transitions (A to F for example) for the same (B to B).
SecurityScorecard: Transition matrix for followed vendors
SecurityScorecard and the stability of security ratings: Key takeaways
SecurityScorecard is a trusted cybersecurity ratings company that provides accurate and stable security scores as supported through transition matrices. Ratings are generally stable. This is represented by the fact that the highest likelihood of any company’s score one year from today is that they will likely retain the same score.
The most followed companies have extremely stable scores because this makes the SecurityScorecard platform more reliable for our customers.
Overall, stability also makes ratings more credible and actionable. When ratings are stable, it means that they are not being manipulated or gamed and that they can be used to make decisions about security risk without having to worry about them changing overnight. In summary, stability helps make ratings more actionable and useful for managing security risk.