Alex Gendler published an interesting TED-Ed video on the false-positive paradox. He narrates a hypothetical scenario where a precious metal ‘unobtainium,’ present within only 1% of rocks in a specific quarry, must be mined. Joe, the miner, uses a reading device to identify which rocks contain the precious metal. His device will always detect unobtanium when present. It will also give the correct reading 90% of the time when a rock does not contain unobtanium. Now say Joe’s device beeps when pointed to a rock, and he offers to sell it to you for $200. Is it worth buying this rock from Joe for $200, knowing that you can sell an unobtanium rock for $1000, based on the reading of the detection device?

To determine the answer to this question, it is imperative to note how rare the presence of unobtanium is as well as the 10% likelihood of false positives from the reading device.



Any anomaly detection test or model that is not 100% accurate bears the risk of false negatives and false positives. In the revenue assurance and fraud management (RAFM) domain, the presence of false positives can be tricky as it utilizes resources to investigate that the anomaly is, in fact, not a problem.

Like Gendler’s scenario, one fundamental point to remember is that revenue leakage within revenue assurance and fraud management departments is quite rare. But since nearly 75% of operators report at least 1% leakage of the total revenue, false positives warrant some investigation.

What would be the successful formula for investing in a system that identifies rare leakages or fraud, especially when false positives abound?

Here are some key elements to look out for:

1. Calculating ROI

Any well-designed revenue assurance and fraud management strategy must articulate short-term as well as long-term benefits. But as time passes, software/controls age, become redundant or lose their effectiveness.

Moreover, it is seen that as operator maturity increases, leakages decrease and, hence, the probability of identifying these leakages.

Recommendations:

From a practical perspective, operators need updated solutions to adapt flexibly to changing scenarios such as new revenue streams and broader risk coverage.

Each solution should encompass well-defined KPIs to measure not only the intended output of revenue assurance and fraud management (like the number of subscribers impacted and revenue at risk) but also its quality (like the effectiveness of revenue assurance and fraud management control).

2. Choosing controls

The process of designing a ‘control’ or a ‘check’ is the key to minimizing the hassle of false positives. Revenue assurance and fraud management teams often struggle to manage the reference data, data specifications, connectivity, latency, etc.

All data management parameters can influence false positives and, thus, require timely correction. Since all control systems are not equal, arriving at an effective control design (including the alarm thresholds) is challenging.

Recommendations:

Establish key indicators to measure system performance for business outcomes as well as revenue assurance and fraud management processes.

Track support processes on overall outcomes to avoid unwanted influences on the quality of revenue assurance and fraud management controls.



Streamline processes through artificial intelligence (AI) and machine learning (ML) to correlate different outcomes and reduce the number of false positives. To understand the significance of this, even low levels of false positives compromise the ability to detect genuine leakages.

3. Maintaining accuracy

The probability of finding a genuine alarm in the revenue assurance and fraud management system indicates how much effort is needed to identify leakages. It also signals the number of alarms that will be closed as ‘non-fraud.’ False positives are usually measured from the system’s perspective, i.e., how many false alarms exist out of the total alarms generated? But most systems do not reflect the actual effort involved in getting the expected benefits of the control.

Recommendations:

Identify the potential trade-offs between implementing a control, its benefits, and its costs. The cost is mostly understood as a development effort but should also include maintenance and monitoring effort.

Consider each control in the broader context of the overarching control framework and risk coverage. This may unearth risks that are over-controlled or under-controlled, thereby suggesting areas that need improvement.

4. Easy monitoring

When the probability of identifying a genuine fraud attempt or revenue leak decreases, the monitoring effort increases because more alarms and controls will need to be set to identify the incident.

Recommendations:

AI and ML can quickly identify false positives, determine likelihoods, and present these to analysts for immediate action.
Revenue assurance managers can save time spent on investigating cases that are no leakages. For fraud management leaders, it helps avoid customer discontent when cases are not fraudulent.

Dealing with the uncertainty of false positives is a painstaking exercise. By implementing the right controls and tweaking these systematically, revenue assurance and fraud management teams can maximize the return on investments of their monitoring systems.



(The answer to the unobtainium question is: No. Statistically, the buyer has only a 9% chance of finding unobtainium in the rock, which is pretty poor odds that the $200 investment will pay off!)

In your experience, what best-practices help deal with false positives?

The article is originally published at Subex.

News From
Subex