Data Mining

John Samuel
CPE Lyon

Year: 2018-2019
Email: john(dot)samuel(at)cpe(dot)fr

Creative Commons License

Data Mining | John Samuel
1

Data Mining

Goals

Data Mining | John Samuel
2

Data Mining

Course Structure:

Data Mining | John Samuel
3

Programming Environment

Course:

Class:

Practical Session:

Data Mining | John Samuel
4

Data Mining

Class Dates
Class 1 6th March
Class 2 6th March
Class 3 3rd April
Class 4 3rd April
Data Mining | John Samuel
5

Data Mining

Practical Sessions

Data Mining | John Samuel
6

Data Mining

Practical Sessions: Jupyter notebooks

Data Mining | John Samuel
7

Data Mining: Jupyter Notebooks

Practical Sessions: Jupyter notebooks

Data Mining | John Samuel
8

Data Mining: Jupyter Notebooks

Practical Sessions: Jupyter notebooks

Data Mining | John Samuel
9

Data Mining: Jupyter Notebooks

Data Mining | John Samuel
10

Data Mining

Practical Sessions: Wikidata (Open Data)

Data Mining | John Samuel
11

1. Lifecycle of data

Lifecycle of Data

  1. Data
  2. Knowledge
  3. Insights
  4. Actions
Data Lifecycle
Data Mining | John Samuel
12

1. Lifecycle of data

1.1. From Data to Knowledge

  1. Data acquisition
  2. Data Extraction
  3. Data Cleaning
  4. Data Transformation
  5. Data analysis modeling
  6. Data Storage
  7. Analysis
  8. Visualisation
Major steps of data analysis
Data Mining | John Samuel
13

1. Lifecycle of data

Different types of data acquistion techniques
Data Mining | John Samuel
14

1. Lifecycle of data

1.1.2. ETL (Extraction Transformation and Loading)

  1. Data Extraction
  2. Data Cleaning
  3. Data Transformation
  4. Loading data to information stores
ETL (Extraction, Transformation and Loading)
Data Mining | John Samuel
15

1. Lifecycle of data

1.1.3. Data analysis
Data Mining | John Samuel
16

1. Lifecycle of data

1.1.4 Data visualization
Data Mining | John Samuel
17

2. Data Acquistion and Storage

2.1. Data acquisition

  1. Surveys
    • Manual surveys
    • Online surveys
  2. Sensors1
    • Temperature, pressure, humidity, rainfall
    • Acoustic, navigation
    • Proximity, presence sensors
  3. Social networks
  4. Video surveillance cameras
  5. Web
Different types of data acquistion techniques
  1. https://en.wikipedia.org/wiki/List_of_sensors
Data Mining | John Samuel
18

2. Data Acquistion and Storage

2.2. Data storage formats

Data Mining | John Samuel
19

2. Data Acquistion and Storage

2.2 Types of data stores

  1. Structured data stores
    • Relational databases
    • Object-oriented databases
  2. Unstructured data stores
    • Filesystems
    • Content-management systems
    • Document collections
  3. Semi-structured data stores
    • Filesystems
    • NoSQL data stores
Unstructured vs. Structured vs. Semi-structured
Data Mining | John Samuel
20

2. Data Acquistion and Storage

2.3.1. ACID Transactions1

  1. https://en.wikipedia.org/wiki/ACID
Data Mining | John Samuel
21

2. Data Acquistion and Storage

2.3.1. ACID Transactions

Data Mining | John Samuel
22

2. Data Acquistion and Storage

2.3.2. Types of data stores

Data Mining | John Samuel
23

2. Data Acquistion and Storage

2.3.3. NoSQL

Data Mining | John Samuel
24

2. Data Acquistion and Storage

2.3.3. Types of NoSQL stores

Data Mining | John Samuel
25

3. Data Extraction and Integration

3.1. Data extraction techniques

Data Mining | John Samuel
26

3. Data Extraction and Integration

3.2. Query interfaces

Data Mining | John Samuel
27

3. Data Extraction and Integration

3.3. Crawlers for web pages

Web crawlers: navigating the entire using hyperlinks
Data Mining | John Samuel
28

3. Data Extraction and Integration

3.4. Application Programming Interface (API)

API (Interface de programmation)
Data Mining | John Samuel
29

4. Pre-treatement of Data

4.1 Data Cleaning: Types of Errors

Data Mining | John Samuel
30

4. Pre-treatement of Data

4.1.1. Syntactical errors

Data Mining | John Samuel
31

4. Pre-treatement of Data

4.1.2. Semantic errors

Data Mining | John Samuel
32

4. Pre-treatement of Data

4.1.3. Coverage errors

Data Mining | John Samuel
33

4. Pre-treatement of Data

4.2.1. Handling Syntactical errors

Data Mining | John Samuel
34

4. Pre-treatement of Data

4.2.2. Handling Semantic errors

Data Mining | John Samuel
35

4. Pre-treatement of Data

4.2.3. Handling Coverage errors

Data Mining | John Samuel
36

4. Pre-treatement of Data

4.2.4. Administrators and handling errors

Data Mining | John Samuel
37

5. Data Transformation

5.1 Languages

Data Mining | John Samuel
38

6. ETL

6.1. ETL (Extraction Transformation and Loading)

  1. Data Extraction
  2. Data Cleaning
  3. Data Transformation
  4. Loading data to information stores
Data Mining | John Samuel
39

6. ETL

6.2.1. Models for data analysis

Data Mining | John Samuel
40

6. ETL

6.2.1. Models for data analysis

Data Mining | John Samuel
41

6. ETL

6.2.3. Star Schema

Data Mining | John Samuel
42

6. ETL

6.2.3. Data Cubes

Data Mining | John Samuel
43

6. ETL

6.2.4. Snow Schema

Data Mining | John Samuel
44

6. ETL

6.2. ETL: From one data store to another

Data Mining | John Samuel
45

7. Data Analysis

Activities of data analysis

  1. Retrieving values
  2. Filter
  3. Compute derived values
  4. Find extremum
  5. Sort
  6. Determine range
  7. Characterize distribution
  8. Find analysis
  9. Cluster
  10. Correlate
  11. Contextualization
  1. https://en.wikipedia.org/wiki/Data_analysis
Data Mining | John Samuel
46

8. Data Visualization

8.1. Data Visualization

  1. Time-series
  2. Ranking
  3. Part-to-whole
  4. Deviation
  5. Sort
  6. Frequency distribution
  7. Correlation
  8. Nominal comparison
  9. Geographic or geospatial
  1. https://en.wikipedia.org/wiki/Data_visualization
Data Mining | John Samuel
47

8. Data Visualization

8.2. Data Visualization: Examples

  1. Bar-chart (Nominal comparison)
  2. Pie-chart (part-to-whole)
  3. Histograms (frequency-distribution)
  4. Scatter-plot (correlation)
  5. Network
  6. Line-chart (time-series)
  7. Treemap
  8. Gantt chart
  9. Heatmap
Data Mining | John Samuel
48

8. Data Visualization

Dominant Colors (Pie Chart)
Data Mining | John Samuel
49

8. Data Visualization

Programming Language Paradigms (Bubble Chart)
Data Mining | John Samuel
50

8. Data Visualization

Timeline of Programming Languages (using Histropedia)
Data Mining | John Samuel
51

8. Data Visualization

Influence Graph of Programming Languages
Data Mining | John Samuel
52

8. Data Visualization

k Predominant colours
Data Mining | John Samuel
53

8. Data Visualization

RGB Scatter plots (Comparison)
Data Mining | John Samuel
54

References

Colors

Images

Data Mining | John Samuel
55