Data Science Process

The Data Science Process


what does a Data Scientist do?”. Or “what does a day in the data science life look like?” These questions are tricky. The answer can vary by role and company.

Here’s a summary of insights.

Step 1: Frame the problem
The first thing you have to do before you solve a problem is to define exactly what it is. You need to be able to translate data questions into something actionable.

You’ll often get ambiguous inputs from the people who have problems. You’ll have to develop the intuition to turn scarce inputs into actionable outputs--and to ask the questions that nobody else is asking.

Say you’re solving a problem for the VP Sales of your company. You should start by understanding their goals and the underlying why behind their data questions.

Before you can start thinking of solutions, you’ll want to work with them to clearly define the problem.

A great way to do this is to ask the right questions.

You should then figure out what the sales process looks like, and who the customers are. You need as much context as possible for your numbers to become insights.

You should ask questions like the following:

1. Who are the customers?
2. Why are they buying our product?
3. How do we predict if a customer is going to buy our product?
4. What is different from segments who are performing well and those that are performing below expectations?
5. How much money will we lose if we don’t actively sell the product to these groups?

In response to your questions, the VP Sales might reveal that they want to understand why certain segments of customers have bought less than expected. Their end goal might be to determine whether to continue to invest in these segments, or de-prioritize them. You’ll want to tailor your analysis to that problem, and unearth insights that can support either conclusion.

It’s important that at the end of this stage, you have all of the information and context you need to solve this problem. 

You need as much context as possible for your numbers to become insights.

Step 2: Collect the raw data needed for your problem

Once you’ve defined the problem, you’ll need data to give you the insights needed to turn the problem around with a solution. This part of the process involves

thinking through what data you’ll need and finding ways to get that data, whether it’s querying internal databases, or purchasing external datasets.

You might find out that your company stores all of their sales data in a CRM or a customer relationship management software platform.You can export the CRM data in a CSV file for further analysis.

Step 3: Process the data for analysis

Now that you have all of the raw data, you’ll need to process it before you can do any analysis. Oftentimes, data can be quite messy, especially if it hasn’t been well-maintained. You’ll see errors that will corrupt your analysis: values set to null though they really are zero, duplicate values, and missing values. It’s up to you to go through and check your data to make sure you’ll get accurate insights.

You’ll want to check for the following common errors:

1. Missing values
2. Corrupted values
3. Timezone differences
4. Date range errors, such as data registered from before sales started

You’ll need to look through aggregates of your file rows and columns and sample some test values to see if your values make sense. If you detect something that doesn’t make sense, you’ll need to remove that data or replace it with a default value. You’ll need to use your intuition here: if a customer doesn’t have an initial contact date, does it make sense to say that there was NO initial contact date? Or do you have to hunt down the VP Sales and ask if anybody has data on the customer’s missing initial contact dates?
Once you’re done working with those questions and cleaning your data, you’ll be ready for exploratory data analysis (EDA).

Step 4: Explore the data

When your data is clean, you’ll should start playing with it!

The difficulty here isn’t coming up with ideas to test, it’s coming up with ideas that are likely to turn into insights. You’ll have a fixed deadline for your data science project (your VP Sales is probably waiting on your analysis eagerly!), so you’ll have to prioritize your questions. 

You’ll have to look at interesting patterns that explain why sales are reduced for this group. You might notice that they don’t tend to be very active on social media, with few of them having Twitter or Facebook accounts. You might also notice

that most of them are older than your general audience. From that you can begin to trace patterns you can analyze more deeply.

Step 5: Perform in-depth analysis

This step of the process is where you’re going to have to apply your statistical, mathematical and technological knowledge and leverage all of the data science tools at your disposal to crunch the data and find every insight you can.

In this case, you might have to create a predictive model that compares your underperforming group with your average customer. You might find out that the age and social media activity are significant factors in predicting who will buy the product.

If you’d asked a lot of the right questions while framing your problem, you might realize that the company has been concentrating heavily on social media marketing efforts, with messaging that is aimed at younger audiences.

You would know that certain demographics prefer being reached by telephone rather than by social media. You begin to see how the way the product has been has been marketed is significantly affecting sales: maybe this problem group isn’t a lost cause! A change in tactics from social media marketing to more in-person
interactions could change everything for the better. This is something you’ll have to flag to your VP Sales.

You can now combine all of those qualitative insights with data from your quantitative analysis to craft a story that moves people to action.


Step 6: Communicate results of the analysis 

It’s important that the VP Sales understand why the insights you’ve uncovered are important. Ultimately, you’ve been called upon to create a solution throughout the data science process. Proper communication will mean the difference between action and inaction on your proposals.

Proper communication will mean the difference between action and inaction on your proposals.

You need to craft a compelling story here that ties your data with their knowledge. You start by explaining the reasons behind the underperformance of the older demographic. You tie that in with the answers your VP Sales gave you and the insights you’ve uncovered from the data. Then you move to concrete solutions that address the problem: we could shift some resources from social media to personal calls. You tie it all together into a narrative that solves the pain of your VP Sales: she now has clarity on how she can reclaim sales and hit her objectives.

She is now ready to act on your proposals.

***

As a data scientist, you’ll have to learn how to work through the entire data science process. Here’s how that looks like from day to day

The Different Data Science Roles

The Different Data Science Roles

Before we dive too deep into what skills you need to become a data scientist,you should be aware that there are different roles in data science. Oftentimes, a data science team will rely on different team members for different skill sets. Or the skill set needed may depend on the type of company and part of the organization you work in. You don’t have to become the world’s best at everything.

While there are some basics every data scientist should know (e.g. basic statistics), data science roles can vary significantly in their demands and expectations. 

Let’s look at the some broad categories of roles that all get lumped under the umbrella term “Data Science”


Data Scientists

One definition of a data scientist is someone who knows more programming than a statistician, and more statistics than a software engineer. Data scientists fine-tune the statistical and mathematical models that are applied onto that data. This could involve applying theoretical knowledge of statistics and algorithms to find the best way to solve a data problem.


Data Analysts and Business Analysts

Data analysts sift through data and provide reports and visualizations to explain what insights the data is hiding. When somebody helps people from across the company understand specific queries with charts, they are filling the data analyst
(or business analyst) role. In some ways, you can think of them as junior data scientists, or the first step on the way to a data science job.

Business analysts are a group that’s adjacent to data analysts, and are more concerned with the business implications of the data and the actions that should result. Should the company invest more in project X or project Y? Business analysts will leverage the work of data science teams to communicate an answer.and visualizations
to explain what insights the data is hiding. When Chase from Higher Learning Technologies helps people from across the company understand specific queries with charts, he is filling the business analyst role.

This blog post summarizes some of the differences. You can roughly say that data engineers rely more on engineering skills, data scientists rely more on their training in mathematics and statistics, and business analysts rely more heavily on their communication skills and their domain expertise. You can be sure that people who occupy these roles will have varying amounts of skills outside of their specialties though and that they can all broadly use the skills we describe below.

It’s important to keep this consideration in mind because data science can be a big tent, and you can pick and choose your spots.

For instance, a data scientist might use historical data to build a model that predicts the number of credit card defaults in the following month.

A data scientist will be able to run with data science projects from end-to-end. They can store and clean large amounts of data, explore data sets to identify insights, build predictive models and weave a story around the findings.

Within the broad category of data scientists, you might encounter statisticians who focus on statistical approaches to data, and data managers who focus on running data science teams.

Data scientists are the bridge between programming and implementation of data

For instance, a data scientist might use historical data to build a model that predicts the number of credit card defaults in the following month.

A data scientist will be able to run with data science projects from end-to-end. They can store and clean large amounts of data, explore data sets to identify insights, build predictive models and weave a story around the findings.

Within the broad category of data scientists, you might encounter statisticians who focus on statistical approaches to data, and data managers who focus on running data science teams.

Data scientists are the bridge between programming and implementation of data science, the theory of data science, and the business implications of data.


Data Engineers

Data engineers are software engineers who handle large amounts of data, and often lay the groundwork and plumbing for data scientists to do their jobs effectively. They are responsible for managing database systems, scaling the data architecture to multiple servers, and writing complex queries to sift through the data. They might also clean up data sets, and implement complex requests that come from data scientists, e.g. they take the predictive model from the data scientist and implements it into production-ready code.

Data engineers, in addition to knowing a breadth of programming languages (e.g. Ruby or Python), will usually know some Hadoop-based technologies (e.g. MapReduce, Hive, and Pig) database technologies like MySQL, Cassandra and MongoDB.

Within the broad category of data engineers, you’ll find data architects who focus on structuring the technology that manages data models and database administrators who focus on managing data storage solutions.


Data Analysts and Business Analysts

Data analysts sift through data and provide reports and visualizations to explain what insights the data is hiding. When somebody helps people from across the company understand specific queries with charts, they are filling the data analyst
(or business analyst) role. In some ways, you can think of them as junior data scientists, or the first step on the way to a data science job.

Business analysts are a group that’s adjacent to data analysts, and are more concerned with the business implications of the data and the actions that should result. Should the company invest more in project X or project Y? Business analysts will leverage the work of data science teams to communicate an answer.


Skills

You can roughly say that data engineers rely most heavily on software engineering skills, data scientists rely on their training in statistics and mathematical modeling, and business analysts rely more heavily on their analytical skills and domain expertise. You can be sure that people who occupy these roles will have varying amounts of skills outside of their specialties.

It’s important to keep this consideration in mind because data science can be a big tent, and you can pick and choose your spots, but each spot comes with different needs, and different salaries.


Salary Ranges

Data scientists need to have the broadest set of skills that covers the theory, implementation and communication of data science. As such, they also tend to be the highest compensated group with an average salary above $115,000 USD.

Data engineers focus on setting up data systems and making sure code is clean, and technical systems are well-suited to the amount of data passing back and forth for analysis. They tend to be middle of the pack when it comes to compensation, with an average salary around $100,000 USD.

Data analysts often focus on querying information and communicating insights found to drive action within organizations. While their average salary is around $65,000 USD, this is partly because a lot of data analyst roles are filled by entry-level graduates with limited work experience.

Every one of these roles combines together into a whole data science team that can solve any data problem placed in front of them.

Getting Your First Data Science Job

Data Science to the Rescue


You’ve probably heard that being a data scientist is the sexiest career of the 21st century, one where you can earn a healthy salary, and a great work-life balance.

Google’s Chief Economist, Hal Varian, has said that, “The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades”.

GiveDirectly is just one example of how organizations win by using data to their advantage.

According to LinkedIn, Statistical Analysis & Data Mining were the hottest skills that got recruiters’ attention in 2014. Glassdoor ranked Data Scientist as the #1 job to pursue in 2016. Some people have even called it the sexiest career of the 21st century.

Google’s Chief Economist, Hal Varian, has said that, “The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades”.

But sending people to each village could take several trips at a crushing expense, creating overheads for an organization looking to operate leanly.


“The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades.” 

- Hal Varian, Google’s Chief Economist

Liaising with GiveDirectly, a pair of industry experts from IBM and Enigma set out to see if data science could help.

Using satellite images provided by Google, they were able to use computers to classify which villages had metal roofs on top of their houses, and which ones had thatch. They were able to determine which villages needed the most help without sending a single person to the area.

This required mining satellite data and making sense of massive amounts of data, something that would have been impossible a decade ago. It required implementing machine learning algorithms, a cutting-edge technology at the time, to train computers to recognize patterns.

These data scientists were able to pinpoint where GiveDirectly should operate, saving the organization hundreds of man-hours and allowing it to do what it does best: solving extreme poverty.


What is Data Science?

GiveDirectly is just one example of how organizations win by using data to their advantage.

Around the world, organizations are creating more data every day, yet most are struggling to benefit from it. According to McKinsey, the US alone will face a shortage of 150,000+ data analysts and an additional 1.5 million data-savvy managers.

According to LinkedIn, Statistical Analysis & Data Mining were the hottest skills that got recruiters’ attention in 2014. Glassdoor ranked Data Scientist as the #1 job to pursue in 2016. Harvard Business Review even called it the sexiest career of the 21st century.
GiveDirectly was able to save thousands of dollars and put their money where their mission is thanks to a team of three data scientists. Within the mass of data the world generates every day, similar insights are hidden away. Each may have the potential to transform entire industries, or to improve millions of lives.

Salary trends have followed the impact data science drives. With a national average salary of $118k (which increases to $126k in Silicon Valley), data science has become a lucrative career path where you can solve hard problems and drive social impact.

Since you’re reading this guide, you’re likely curious about a career in Data Science, and you’ve probably heard some of these facts and figures. You likely know that data science is a career where you can do good while doing well.

You’re ready to dig beyond the surface, and see real-life examples of data science, and get real-life advice from practitioners in the field.

That’s exactly why we wrote this guide. To bring data science careers to life, for thousands of data-curious, savvy young professionals. We hope that after reading this guide, you have a solid understanding of the data science industry, and know what it takes to get your first data science job. We also want to leave you
with a checklist of actionable advice which will help you throughout your data science career.


The Foundations of Data Science

DJ Patil, the current Chief Data Scientist of the United States and previously the Head of Data Products at Linkedin, is the one who first coined the term data science.

A decade after it was first used, the term remains contested. There is a lot of debate among practitioners and academics about what data science means, and whether it’s different at all from the data analytics that companies have always done.

One of the most substantive differences is the amount of data you have to process now as opposed to a decade ago. In 2020, the world will generate 50x more data than we generated in 2011. Data science can be considered an interdisciplinary solution to the explosion of data that takes old data analytics approaches, and uses machines to augment and scale their effects on larger data sets.

DJ posits that, “the dominant trait among data scientists is an intense curiosity—a desire to go beneath the surface of a problem, find the questions at its heart, and distill them into a very clear set of hypotheses that can be tested.” There is no mention here of a strict definition of data science, nor of a profile that must fit it.

Baseball players used to be judged by how good scouts thought they looked, not how many times they got on base--that was until the Oakland A’s won an all-time league record 20 games in a row with one of the lowest paid rosters in the league. Elections used to swing from party to party with little semblance of predictive accuracy--that was until Nate Silver correctly predicted every electoral vote in the 2012 elections.

Data, and a systematic approach to uncover truths about the world around us, have changed the world.

“More than anything, what data scientists do is make discoveries while swimming in data. It’s their preferred method of navigating the world around them,” concludes Patil.

To do data science, you have to be able to find and process large datasets. You’ll often need to understand and use programming, math, and technical communication skills.

Most importantly, you need to have a sense of intellectual curiosity to understand the world through data, and not be deterred easily by obstacles.

You might not think you know anything about data science, but if you’ve ever looked for a Wikipedia table to settle a debate with one of your friends, you were doing a little bit of data science.