Model Performance with Yellowbrick
Model Performance with Yellowbrick, Detecting healthcare fraud is a complex endeavor due to the inherent class imbalance present in claims data.
In previous discussions, we explored how Yellowbrick’s Class Balance visualizer aids in understanding and addressing this imbalance between fraudulent and non-fraudulent providers.
Model Performance with Yellowbrick
We also examined resampling techniques like SMOTE to enhance our model’s capability in detecting potential fraud cases effectively.
The Importance of Feature Engineering
Feature engineering plays a pivotal role in fraud detection modeling. By aggregating individual claims at the provider level, we can reveal distinctive patterns indicative of fraudulent behavior.
Using Yellowbrick’s Parallel Coordinates visualizer, we identified combinations of inpatient and outpatient claims that might suggest fraudulent activities.
This process transforms granular transaction data into meaningful insights at the provider level.
Evaluating Model Performance
With our data now preprocessed and informative features engineered, it’s time to evaluate how well our model can differentiate between fraudulent and legitimate providers.
To achieve this, we will utilize Yellowbrick’s Confusion Matrix visualizer, which provides a deeper understanding of model performance beyond mere accuracy metrics.
Our dataset comprises 5,410 healthcare providers, featuring more than 550,000 individual claims records. For detailed steps on obtaining our provider-level dataset from the original claims data, refer to our earlier posts:
Here’s a snapshot of our dataset (previously titled ‘final_df’):
Provider IP_Claims_Total OP_Claims_Total PotentialFraud
0 PRV51001 97000.0 7640.0 No
1 PRV51003 573000.0 32670.0 Yes
2 PRV51007 19000.0 14710.0 No
3 PRV51008 25000.0 10630.0 No
4 PRV51011 5000.0 11630.0 No
... ... ... ... ...
5405 PRV57759 0.0 10640.0 No
5406 PRV57760 0.0 4770.0 No
5407 PRV57761 0.0 18470.0 No
5408 PRV57762 0.0 1900.0 No
5409 PRV57763 0.0 43610.0 No
[5410 rows x 4 columns]Data Preprocessing and Target Encoding
Before we can begin training our model, we need to encode our target variable. The LabelEncoder transforms our categorical fraud labels (‘Yes’ and ‘No’) into binary values (1 for fraud, 0 for non-fraud):
from sklearn.preprocessing import LabelEncoder
# Encode the PotentialFraud column as a binary variable
le = LabelEncoder()
final_df["PotentialFraud"] = le.fit_transform(final_df["PotentialFraud"])After encoding, our dataset looks as follows:
Provider IP_Claims_Total OP_Claims_Total PotentialFraud
0 PRV51001 97000.0 7640.0 0
1 PRV51003 573000.0 32670.0 1
2 PRV51007 19000.0 14710.0 0
3 PRV51008 25000.0 10630.0 0
4 PRV51011 5000.0 11630.0 0
... ... ... ... ...
5405 PRV57759 0.0 10640.0 0
5406 PRV57760 0.0 4770.0 0
5407 PRV57761 0.0 18470.0 0
5408 PRV57762 0.0 1900.0 0
5409 PRV57763 0.0 43610.0 0
[5410 rows x 4 columns]Building Our Initial Model
With our data prepared, we can now train a logistic regression model using our claims totals as features:
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
# Define features and target variable
X = final_df[["IP_Claims_Total", "OP_Claims_Total"]]
y = final_df["PotentialFraud"]
# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Split the dataset into training and testing sets (80-20 split)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
# Train a logistic regression model
log_reg = LogisticRegression(random_state=42)
log_reg.fit(X_train, y_train)
# Predictions
y_pred = log_reg.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"**Accuracy:** {accuracy:.4f}\n")The initial model achieves an impressive accuracy of 93.25%. However, accuracy alone doesn’t present the full picture, especially for imbalanced datasets.
This is where Yellowbrick’s Confusion Matrix visualizer proves invaluable.
Visualizing Model Performance with Confusion Matrix
Yellowbrick facilitates the creation of a clear and insightful confusion matrix visualization. With just a few lines of code, we can represent our model’s predictions in an easily interpretable format:
from yellowbrick.classifier import ConfusionMatrix
cm = ConfusionMatrix(log_reg, classes=['No Fraud', 'Fraud'])
cm.fit(X_train, y_train)
cm.score(X_test, y_test)
cm.finalize()The confusion matrix provides a breakdown of predictions into key components:
- True Negatives (969): Correctly identified non-fraudulent providers
- True Positives (40): Correctly identified fraudulent providers
- False Negatives (64): Fraudulent providers misclassified as non-fraudulent
- False Positives (9): Non-fraudulent providers misclassified as fraudulent
Although our accuracy score of 93.25% is commendable, it obscures critical insights regarding our model’s performance, particularly concerning recall for fraudulent cases.
From 104 actual fraud cases (40 true positives and 64 false negatives), our model only identified 40, resulting in a recall of only 38.5% (40/104).
This finding is alarming in a healthcare fraud detection context, where missing fraudulent cases can result in substantial financial losses.
Conclusion
Yellowbrick’s Confusion Matrix visualizer is an essential tool for translating model performance into clear, interpretable results.
It integrates seamlessly with scikit-learn, helping data scientists understand their models beyond basic accuracy scores.
In our healthcare fraud detection analysis, the visualizer illuminated significant gaps in fraud identification, aspects that accuracy metrics alone failed to capture.
This emphasizes the importance of utilizing comprehensive model evaluation tools in healthcare fraud detection.
