8 483 ₽/month
Read more
Group 1321314279 (1)
Data Analyst
Why and from what data is cleared
Imagine that you need to find an answer to a question online. But half of the articles in the search are on a different topic, and many of the information is outdated or incorrect. In such circumstances, it will be difficult to find the correct answer.
Models in Data Science and Machine Learning face the same problem. The data they are trained on can contain a lot of “garbage”: incorrect values, errors, duplicates. This occurs because information is usually collected from many different sources, each with its own representation of the data. Because of this, the data in the sample is heterogeneous and sometimes incorrect.
The model can learn on dirty data, but this can greatly reduce its accuracy. If you don’t clean the data before loading it into the model, there’s a high risk that it will end up producing incorrect results — say, forecasts that are far from the truth.
Therefore, in order for the model to work accurately, the data needs to be cleared of “garbage” before training it:
remove errors and inconsistencies that occur in the data sample;
bring data to a unified form, for example, combine identical features;
fill in missing values, remove duplicates;
get rid of noise and outliers - random values that differ sharply from the majority.
Read also
Alisa Radchenko: “I worked as an accountant, and now bangladesh telegram data I analyze data at MTS”
What are the types of errors in data?
Usually, information is stored in special storage facilities - databases . They can be arranged in different ways, but most often, entities in databases can be divided into two categories:
records - rows in a table, some objects that consist of a set of features;
features are values in table cells that describe some characteristics of an object.
For example, we have a record about the user misha. This record is a row in a table that contains all the attributes of the user misha. The attributes can be a nickname, age, gender, activity data, and so on. Together, they make up the record.
Data errors may be attribute-specific or record-specific. For each category, several common types of "contamination" are distinguished.
In fact, there are more possible problems than described in the article. We have only discussed those that occur most often. Source
Errors in records. At the level of the entire record, four types of errors can occur: