Architecture of Information Systems
Search Engine

John Samuel
CPE Lyon

Year: 2017-2018
Email: john(dot)samuel(at)cpe(dot)fr

Outline: Search engine

Frontend development
Backend development
Application programming interface

Regular users
Domain Experts

Search interface

One-box search
Advanced search (filters)

Personalized user experience

Queries

Keywords
Natural language queries

Search Results

Links
Structured response
Natural language response

Leonardo da Vinci (October 2017 Google results)

Filter search results (Multiple boxes)

Location of Archaelogical sites (Wikidata)

Why filters?

Reduce information overload
Precise queries
Interactive search

Operators

AND
OR
NOT

Mnemonics

Time and location (Internationalization)

Past user search queries

Data collection
Data storage
Configuration
Logging
Dashboard
Security

Data ownership

Internal data (e.g., official website, internal wikis, databases etc.)
External data (e.g., other websites, wikis, open data)

Data model (Data and Schema)

Unstructured data (e.g., documents, texts, web pages etc.)
Semi-structured data (e.g., JSON/XML files etc.)
Structured data (e.g., relational databases, linked data)

Data sources

Web pages
Documents, texts
Sensors
Databases
...

Data acquisition

Data dumps
Crawlers
Web scraping
Application Programming Interface (API)

import requests
url = "https://api.github.com/"

response = requests.get(url)
print(response.json())

from SPARQLWrapper import SPARQLWrapper, JSON

sparql = SPARQLWrapper("http://query.wikidata.org/sparql")
sparql.setQuery("""
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q9143;
}
LIMIT 10
""")
sparql.setReturnFormat(JSON)
results = sparql.query().convert()

for result in results["results"]["bindings"]:
    print(result)

Data cleaning and transformation

Accuracy (e.g., verification with external sources)
Validity (e.g., detect constraint violations)
Uniformity (e.g., units)

Model
Indexation
Query optimization
Caching
Replication
Backup

Database schema
Schema-less

Relational Databases
Object-oriented Databases
NoSQL databases (e.g., graph databases)
NewSQL databases (SQL + ACID guarantees)

Document indices

Forward index
Inverted index

Database Indexation

Query Optimization

Join ordering
Cost estimation

Frequently asked questions and cached responses

Machines (servers, disks etc.)
Software packages and dependencies
Energy consumption

Development setup
Pre-production setup
Production setup

Containers (e.g., Linux containers, Docker)

Server-side
Client-side

A/B Testing

Access logs
Error logs
Event logs
Transaction logs

Why logs?

Debugging
Security (e.g., detect intrusion)
Database rollbacks
Audit
Analysis (e.g., detecting patterns, resource planning)

IP address
User
Resource ID
...

Wikimedia (Availability: 5^th October 2017)

Performance Metrics (active users, queries served etc.)
Real-time metrics (e.g., downtime, latency, throughput)

Email Alerts
Visual indicators

Data protection
Logged-in users or public access
Third party access

Authentication
Authorization to third party access

Intrusion
SQL code injection
Cross-site scripting
Denial of service

Service-oriented (SOAP)
Resource-oriented (REST)

XML
JSON
HTML?

Create
Read
Update
Delete
List

Complete data dumps
Selective data dumps

HTTP
Software development kits (SDKs)

Human-readable documentation
Machine-readable documentation (WSDL, WADL etc.)
Human and machine-readable documentation (microformats, semantic web languages)

Read documentation
Develop application to integrate
Add business logic, if any

Fully autonomous solution to integrate
Add business logic, if any

Resource usage limits

Limits on API call count (per user, IP)
Limits on data transfer
Temporary blocks

Analysis on frequently made API calls
Resource planning and allocation

No password
Basic authentication (e.g., username, password)
(Open) authentication protocols (e.g., OAuth)

Architecture of Information Systems Search Engine

Architecture of Information Systems

Outline: Search engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine

1. Frontend development: Search Engine Filters

1. Frontend development: Search Engine Filters

1. Frontend Development: Target Audience

1. Front-end development

Search interface

Personalized user experience

1. Frontend development: Simple search (One box)

Queries

Search Results

1. Frontend development: Advanced search (filters)

Filter search results (Multiple boxes)

1. Frontend development: Advanced search (filters)

1. Frontend development: Advanced search (filters)

1. Frontend development: Advanced search (filters)

1. Frontend development: Advanced search (filters)

Why filters?

1. Frontend development: Advanced Search in one-box

Operators

1. Frontend development: Advanced Search in one-box

Mnemonics

1. Frontend development: Personalized user experience

Time and location (Internationalization)

1. Frontend development: Personalized user experience

Past user search queries

2. Backend development

2.1 Backend development: Data collection

Data ownership

Data model (Data and Schema)

2.1 Backend development: Data collection

Data sources

2.1 Backend development: Data collection

Data acquisition

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

2.1 Backend development: Data collection

Data cleaning and transformation

2.2 Backend development: Data storage

2.2. Backend development: Data Model

2.2 Backend development: Data storage

2.2. Document indices and Query Optimization

Document indices

Database Indexation

Query Optimization

2.2. Backend development: Caching

2.2. Backend development: Replication and Backup

2.3. Resource management and configuration

2.3. Resource management and configuration

2.3. Deployment

2.3. Packaging

2.3. Load balancing

2.3. Selective Testing

A/B Testing

2.4. Backend development: Logging

2.4. Logging

Why logs?

2.4. Logging

2.5. Backend development: Dashboard

2.5. Backend development: Dashboard

2.5. Backend development: Dashboard

2.5. Backend development: Dashboard

2.6. Backend development: Security

2.6. Backend development: Security

2.6. Detecting security vulnerabilities

3. Application programming interface

3. API: Data formats

3. API: (CRUDL) Operations

3. API: Examples

3. API: Examples

3. API: Examples

3. API: Examples

Architecture of Information Systems
Search Engine