Architecture of Information Systems
Search Engine

John Samuel
CPE Lyon

Year: 2018-2019
Email: john(dot)samuel(at)cpe(dot)fr

Creative Commons License

Architecture of Information Systems | John Samuel
1

Architecture of Information Systems

Outline: Search engine

  1. Frontend development
  2. Backend development
  3. Application programming interface
  4. Towards Federated Search Engines
Architecture of Information Systems | John Samuel
2

1. Frontend development: Search Engine

Target devices

Architecture of Information Systems | John Samuel
3

1. Frontend development: Search Engine

Desktop PC

DuckDuckGo
Architecture of Information Systems | John Samuel
4

1. Frontend development: Search Engine

Desktop PC

Qwant
Architecture of Information Systems | John Samuel
5

1. Frontend development: Search Engine

Desktop PC

Wikipedia
Architecture of Information Systems | John Samuel
6

1. Frontend development: Search Engine

Mobile phones

DuckDuckGo
Architecture of Information Systems | John Samuel
7

1. Frontend development: Search Engine

Mobile phones

Google
Architecture of Information Systems | John Samuel
8

1. Frontend development: Search Engine

Mobile phones

Wikipedia
Architecture of Information Systems | John Samuel
9

1. Frontend development: Search Engine

Mobile phones

Wikipedia
Architecture of Information Systems | John Samuel
10

1. Frontend development: Features

Autocompletion

Wikipedia Autocomplete(November 2018)
Architecture of Information Systems | John Samuel
11

1. Frontend development: Features

Autocorrection

Softwre (November 2018 Qwant Autocorrect)
Architecture of Information Systems | John Samuel
12

1. Frontend development: Features

Autocorrection

Softwre (November 2018 DuckDuckGo)
Architecture of Information Systems | John Samuel
13

1. Frontend development: Search Engine

Advanced Search

Architecture of Information Systems | John Samuel
14

1. Frontend development: Search Engine Filters

Advanced Search: Filters

More options (filters)
Architecture of Information Systems | John Samuel
15

1. Frontend development: Search Engine Filters

Advanced Search: Filters

More options (filters)
Architecture of Information Systems | John Samuel
16

1. Frontend development: Search Engine Filters

Advanced Search: Filters

More options (filters)
Architecture of Information Systems | John Samuel
17

1. Frontend Development: Target Audience

Users of search interfaces

Architecture of Information Systems | John Samuel
18

1. Frontend Development: Target Audience

Users of search interfaces

Architecture of Information Systems | John Samuel
19

1. Frontend development: Results

Search Results



Leonardo da Vinci (November 2018 Qwant results)
Architecture of Information Systems | John Samuel
20

1. Frontend development: Results

Search Results



Leonardo da Vinci (October 2017 Google results)
Architecture of Information Systems | John Samuel
21

1. Frontend development: Results

Search Results

Leonardo da Vinci (October 2017 Google results)
Architecture of Information Systems | John Samuel
22

1. Frontend development: Results

Search Results: Timeline

Artists on Histropedia
Architecture of Information Systems | John Samuel
23

1. Frontend development: Results

Search Results: Map

Location of Archaelogical sites (Wikidata)
Architecture of Information Systems | John Samuel
24

1. Frontend development: Results

Search Results: Map

Need for Geographical Coordiantes

SELECT ?item ?location
WHERE {
  ?item wdt:P1435 wd:Q9259;
        wdt:P625 ?location
}
Location of Archaelogical sites (Wikidata)
Architecture of Information Systems | John Samuel
25

1. Frontend development: Results

Search paradigms [1]

Architecture of Information Systems | John Samuel
26

1. Frontend development: Results

Computational knowledge engine



Wolfram Alpha
Architecture of Information Systems | John Samuel
27

1. Frontend development: Results

Search user interfaces

Architecture of Information Systems | John Samuel
28

1. Frontend development: Results

Search Results



Leonardo da Vinci (October 2017 Google results)
Architecture of Information Systems | John Samuel
29

1. Front-end development

Search interface

Personalized user experience

Architecture of Information Systems | John Samuel
30

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

Architecture of Information Systems | John Samuel
31

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

Architecture of Information Systems | John Samuel
32

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

Architecture of Information Systems | John Samuel
33

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

Why filters?

Architecture of Information Systems | John Samuel
34

1. Frontend development: Advanced Search in one-box

Power users

Operators

Architecture of Information Systems | John Samuel
35

1. Frontend development: Advanced Search in one-box

Power users: Mnemonics

Bangs (DuckDuckGo)
Architecture of Information Systems | John Samuel
36

1. Frontend development: Personalized user experience

Context: Time and location (Internationalization)

Weather (weather.com)
Architecture of Information Systems | John Samuel
37

1. Frontend development: Personalized user experience

Past user search queries

User privacy
Architecture of Information Systems | John Samuel
38

2. Backend development

  1. Data collection
  2. Data storage
  3. Configuration
  4. Logging
  5. Dashboard
  6. Security
Architecture of Information Systems | John Samuel
39

2.1 Backend development: Data collection

Data ownership

Data model (Data and Schema)

Architecture of Information Systems | John Samuel
40

2.1 Backend development: Data collection

Data sources

Architecture of Information Systems | John Samuel
41

2.1 Backend development: Data collection

Data acquisition

Architecture of Information Systems | John Samuel
42

2.1 Backend development: Data collection

Architecture of Information Systems | John Samuel
43

2.1 Backend development: Data collection

API
import requests
url = "https://api.github.com/"

response = requests.get(url)
print(response.json())
Architecture of Information Systems | John Samuel
44

2.1 Backend development: Data collection

Open Data
from SPARQLWrapper import SPARQLWrapper, JSON

sparql = SPARQLWrapper("http://query.wikidata.org/sparql")
sparql.setQuery("""
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q9143;
}
LIMIT 10
""")
sparql.setReturnFormat(JSON)
results = sparql.query().convert()

for result in results["results"]["bindings"]:
    print(result)
Architecture of Information Systems | John Samuel
45

2.1 Backend development: Data collection

Linked Open data cloud
Architecture of Information Systems | John Samuel
46

2.1 Backend development: Data collection

Archived and Historical Data
Architecture of Information Systems | John Samuel
47

2.1 Backend development: Data collection

Data cleaning and transformation

Architecture of Information Systems | John Samuel
48

2.2 Backend development: Data storage

Architecture of Information Systems | John Samuel
49

2.2. Backend development: Data Model

Architecture of Information Systems | John Samuel
50

2.2 Backend development: Data storage

Architecture of Information Systems | John Samuel
51

2.2. Document indices and Query Optimization

Document indices

Database Indexation

Query Optimization

Architecture of Information Systems | John Samuel
52

2.2. Backend development: Caching

Architecture of Information Systems | John Samuel
53

2.2. Backend development: Replication and Backup

Replication(Master-slave)
Architecture of Information Systems | John Samuel
54

2.3. Resource management and configuration

Availability (Wikipedia)
Architecture of Information Systems | John Samuel
55

2.3. Resource management and configuration

Architecture of Information Systems | John Samuel
56

2.3. Deployment

Architecture of Information Systems | John Samuel
57

2.3. Packaging

Architecture of Information Systems | John Samuel
58

2.3. Load balancing

Architecture of Information Systems | John Samuel
59

2.3. Selective Testing

A/B Testing

Architecture of Information Systems | John Samuel
60

2.4. Backend development: Logging

Architecture of Information Systems | John Samuel
61

2.4. Logging

Why logs?

Architecture of Information Systems | John Samuel
62

2.4. Logging

Architecture of Information Systems | John Samuel
63

2.5. Backend development: Dashboard

Wikimedia (Grafana: 5th October 2017)
Architecture of Information Systems | John Samuel
64

2.5. Backend development: Dashboard

Wikimedia (Availability: 5th October 2017)
Architecture of Information Systems | John Samuel
65

2.5. Backend development: Dashboard

Architecture of Information Systems | John Samuel
66

2.5. Backend development: Dashboard

Architecture of Information Systems | John Samuel
67

2.6. Backend development: Security

Login (Wikipedia)
Architecture of Information Systems | John Samuel
68

2.6. Backend development: Security

OpenID
Mozilla Persona (2011-2016)
Architecture of Information Systems | John Samuel
69

2.6. Detecting security vulnerabilities

Architecture of Information Systems | John Samuel
70

3. Application programming interface

Architecture of Information Systems | John Samuel
71

3. API: Data formats

Architecture of Information Systems | John Samuel
72

3. API: (CRUDL) Operations

Architecture of Information Systems | John Samuel
73

3. API: Examples

Architecture of Information Systems | John Samuel
74

3. API: Examples

GitHub API: Repository Search
Architecture of Information Systems | John Samuel
75

3. API: Examples

GitHub API: Pagination
Architecture of Information Systems | John Samuel
76

3. API: Examples

Architecture of Information Systems | John Samuel
77

3. API: Data dumps

Architecture of Information Systems | John Samuel
78

3. Application programming interface

Architecture of Information Systems | John Samuel
79

3. Interface definition

Architecture of Information Systems | John Samuel
80

3. Human readable Documentation

  1. Read documentation
  2. Develop application to integrate
  3. Add business logic, if any
Architecture of Information Systems | John Samuel
81

3. Machine-readable Documentation

  1. Fully autonomous solution to integrate
  2. Add business logic, if any
Architecture of Information Systems | John Samuel
82

3. Quality of service

Resource usage limits

Architecture of Information Systems | John Samuel
83

3. Quality of service

Architecture of Information Systems | John Samuel
84

3. Security

OAuth
Architecture of Information Systems | John Samuel
85

4. Towards Federated Search

Federated Search

Architecture of Information Systems | John Samuel
86

Project: Federated Search

Virtual Library

Virtual Library
Architecture of Information Systems | John Samuel
87

Project: Federated Search

Virtual Library: 4 search engines

Architecture of Information Systems | John Samuel
88

Project: Federated Search

Task 1: identify your target audience

Target audience
Architecture of Information Systems | John Samuel
89

Project: Federated Search

Task 2: Build your search interface

Architecture of Information Systems | John Samuel
90

Project: Federated Search

Task 3: Build your results interface

Architecture of Information Systems | John Samuel
91

Project: Federated Search

Task 4: Infrastructure using message queue

Architecture of Information Systems | John Samuel
92

References

References

  1. Rahman, Mahmudur. “Search Engines Going beyond Keyword Search: A Survey.” International Journal of Computer Applications, vol. 75, no. 17, Aug. 2013, pp. 1–8. DOI.org (Crossref)
  2. Marchionini, Gary. “Exploratory Search: From Finding to Understanding.” Communications of the ACM, vol. 49, no. 4, Apr. 2006, p. 41.
  3. Flickner, M., et al. “Query by Image and Video Content: The QBIC System.” Computer, vol. 28, no. 9, Sept. 1995, pp. 23–32.
Architecture of Information Systems | John Samuel
93

References

References

Image credits

Architecture of Information Systems | John Samuel
94