In math we trust: the analytical secrets of credit card companies
I have a heightened sensibility when it comes to recognizing the use of data mining and analytics. Having studied computer science in college, I am fairly knowledgeable of those disciplines. So it was no coincidence that I was able to draw attention to the ill-conceived data mining practices of American Express. Ultimately, my public relations salvo against American Express was not about me being a victim; instead, it was about the credit card company understanding the need to change its algorithm.
I am not sure whether American Express made any adjustments as a result of my efforts and the ensuing public outrage—it said it did—but I do believe that all credit card companies learned a sobering lesson: Mathematical models built on dubious assumptions can and do backfire. In fact, one of the reasons I think my story was so popular—and continues to be today—is that the current economic crisis occurred, in large part, due to our increased reliance on and faith in esoteric and erroneous mathematical models.
During the height of my notoriety, I received hundreds of e-mails from concerned consumers, asking me and giving me insider information about how these models and algorithms work. I responded to each inquiry and often posted these responses in detail on my blog. However, I have never delved into the mathematical concepts of data analysis with an emphasis on how a credit card company might use them. Well, not until now.
My desire to explain some basic methodologies and models on this blog comes from the realization that we live in an increasingly data-driven society. From internet searches to phone records, from credit card purchases to tax returns, companies have access to huge data sets which enable them to, in many cases, know us better than we know ourselves. The access to and interpretation of such data empowers companies beyond belief. This tremendous power should go neither unchecked nor overlooked, especially by the very consumers who are often the subjects of its manipulation.
The next few posts will introduce you to the world of analytics. They will give you confidence to challenge its applications and assumptions. But most importantly, they will help you gain more control over your destiny, which is increasingly determined by the use of mathematical formulas.