Market Segmentation with Python and Modern AI

Welcome back to my blog, glad to have you here! Last time, I introduced the audience to the basics of AI from a high-level (conceptual) point of view, and provided a low-level implementation through detecting focal points of a face (if you haven't read it, you can find it linked at the bottom of this post). I included tons of explanations on AI to gain a better understanding of the power it holds and an entire new problem domain that exists within computer science and software development. This time, I'm using the same technology to tackle a new problem.

Now, let's begin.

* Figure 1: Market Segmentation Illustration

What can you Expect in this Article?

To quickly summarize what to expect, in this article I am going to use Python and AI to perform market segmentation on provided data-sets. Before I explain exactly what market segmentation is, what the benefits are, and how this provides value to companies and individuals from all backgrounds - I want to talk about the feedback I received from the last post and what I will be doing differently in this post.

The feedback I received included the following: that there was simply too-much-information to process, code was too complicated and underexplained, information should be simplified to be better understood. In response, I am going to make this post much more straight-forward, simplified, and provide a clear and concise target for all readers to achieve.

This post will be structured in a step-by-step format so that all readers can achieve the ultimate objectives of this post, which are:

1. Understanding Market Segmentation, how it brings value, and where it can be used.

2. Using Python to import, process and produce sub-markets (segments) from a large pool of data.

3. Analyze sub-markets to produce insights and strategies beneficial to the situation.

What is Market Segmentation and Why Should I Care?

Market segmentation sits within the domain of marketing and is the act of dividing a specific market into smaller sub-groups. If you take a look at figure #1, you will see that the pie-chart is cut into smaller pieces. This is an incredibly beneficial process that allows marketers and executives to gain insights into what types of consumers they have and their respective behaviors.

Market segmentation realizes that not all customers have the same interests, purchasing power, or consumer needs. Instead of catering to all prospective clients broadly, market segmentation is important because it strives to make a company's marketing endeavors more strategic and refined. By developing specific plans for specific products with target audiences in mind, a company can increase its chances of generating sales and being more efficient with resources. [1]

Marketing is crucial to the growth and sustainability of any business. Great marketing includes Identifying consumer' needs and to how to provide a better experience for them. Ultimately, this will drive sales and interactions and lead to a better product overall. The bottom line is that marketers help build the company's brand, engage customers, increase sales, and grow revenue!

Great... but how can I use this? (Examples)

Now that we've gone over the definition and overview of what market segmentation is, it's time to actually apply context this topic. As a software-developer, how can YOU actually benefit from this? I want to help broaden your imagination by introducing some case-studies of where market segmentation is used as a component/feature of a broader system. Most of us interact with targeted market segmentation on a daily basis.

Note: With both of these examples, each company has put a slight twist onto the concept of market segmentation, however, in-essence, they are the same thing; big-data and market segmentation.

• Example 1: Amazon Recommendations

* Figure 2: Amazon Store ("buy it with" feature).

In this example, we can see that as I browse and navigate to a specific product, Amazon begins to recommend additional products that work with my item of choice. The recommendations that it has automatically generated all work together as separate components. More importantly, these additional products match the style, type, and high-performance quality from the base product that I originally selected.

Keep in mind, that Amazon has hundreds of thousands of products available, yet, for each product you select, it will recommend auxiliary "sister" products. It would be impossible for them to do this using a manual approach, the only way they have accomplished this is by using big-data and market-segmentation based on the original product selected. This feature alone generates Amazon millions and is a great example of how market-segmentation can be put into-effect.

Personally, realizing this has made me aware that this concept can be applied to almost ANY business model where there is a demand for something!

• Example 2: Netflix Recommendations

* Figure 3: Netflix Recommendations

The next example is something that almost all of us will be familiar with, Netflix! If you take a closer look at the image above, you can see that Netflix begins to recommend titles based off what you personally are interested in. They do so, by grouping individual titles with data collected from all other users. They find correlations and group them together into common interests.

Key Points:

1. A market can include anything where there is both supply and demand, it does not have to be a physical object.

2. Any place where there are users and data, market segmentation can be applied to enhance the user experience.

3. Market segmentation and big-data are closely interlinked.

Programming with Python!

* Figure 4: Correlation Matrix

We are now on step two, we will be starting off with Python and Google Collab to work with our example. In this project we have been given a data-set that includes purchase data from an online shop that deals with selling accessories for a wide range of vehicles (Motorcycles, Planes, Ships, Trucks, Vintage Cars). The data-set is provided on Kaggle, click HERE to download it. My google Collab link can be accessed by clicking HERE.

Note: All code has been included inside of the Google Collab link for you to view, I will only be providing brief documentation here on how to get started importing the above provided data-set in your own environment. A video has been included to go over the code.

Step 1: Import Libraries and Initialize Data

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import zipfile
import cv2
import plotly.express as px
import tensorflow as tf
from tensorflow.python.keras import Sequential
from tensorflow.keras import layers, optimizers
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, Dropout
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.initializers import glorot_uniform
from tensorflow.keras.utils import plot_model
from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint, LearningRateScheduler
from IPython.display import display
from tensorflow.keras import backend as K
from sklearn.preprocessing import StandardScaler, normalize
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score
from sklearn.manifold import TSNE
from sklearn.metrics.pairwise import cosine_similarity
import plotly.express as px
import plotly.graph_objects as go
from google.colab import files #library to upload files to colab notebook
%matplotlib inline

Step 2: Upload the data-set to your Google Drive

Step 3: Mount the drive and import data-set

# Mount the drive
from google.colab import drive
drive.mount('/content/drive')
sales_df = pd.read_csv('./sales_data_sample.csv', encoding = 'unicode_escape')

Theory behind K-Means Clustering and Centroids

* Figure 5: K-Means algorithm

This concept is critical to understanding the science behind segmentation. K-means is an unsupervised learning algorithm and it works by grouping some data points together in an unsupervised fashion. Taking a look at the figure 5, we can see that we have plotted a graph; the level of education on the x-axis, and on the y-axis, the number of years of experience. When we say unsupervised in this situation, this means that we absolutely do not care about what we are looking for or what the data represents. All we want to do is group the points above into different groups based on proximity and distance from each other.

In the illustration above, we see that after applying the K-Means algorithm to the original data-plot, the points have been grouped into differently colors circles. We refer to these grouping as "clusters". The K-Means algorithm includes the following steps:

1. Choose number of clusters "K".

2. Select random K points that are going to be centroids for each cluster.

3. Assign each data point to the nearest centroid, doing so will enable us to create "K" number of clusters.

4. Calculate a new centroid for each cluster.

5. Reassign each data point to the new closest centroid.

6. Go to step 4 and repeat.

Note: Algorithm steps taken from THIS resource.

The term "centroid" is essentially a focal point of where an individual cluster exists within. The cluster surrounds the centroid and all the points within it. We start off by placing K centroids for each cluster, then assigning each data point to the nearest centroid. If there are any correlations between points of a centroid, then we calculate a new position for the centroid and move it until there are less correlations between centroids. We keep doing this for large data-sets until we have minimal correlations between clusters.

To visually demonstrate this into effect, take a look at the illustration below:

* Figure 6: Reassigning centroids based on proximity

However, for large data-sets, we run into the question of how many individual clusters should we have in the first place? This can be solved through automated testing by introducing an incremental number of centroids and checking to see how many correlations we have between clusters. One important variable to consider is that we only care about generalized clusters, we don't want to overfit and produce an overly sophisticated segmentation of the market.

* Figure 7: Elbow Method

This is called the "elbow method" because we see that as we add additional clusters, the number of correlations (y-axis) begin to drop. However, we only select the optimal number within the market, as introducing additional segments with minimal changes in cluster size will only add to the level of sophistication of our market analysis. Think of it like this, rather than trying to analyze the difference between 10 different segments of the same market, we should only really care about the three biggest differences between our customers.