# Archived: ISP Go Content Filtering Platform (Sales Discontinued)

# 1. Architecture and system requirements

ISP Go solution for Internet Service Providers consists of three components  
• Filtering DNS Server  
• Web application ‘Block page’  
• Web application ‘ISP Go API’ to control and manage user’s settings and integrate the whole ISP Go solution with ISP’s systems.

There are dependencies on the external software packages exist:  
• <span style="color: #ba372a;">`Nginx`</span> is used as a reverse proxy for the block page and ISP Go API  
• <span style="color: #ba372a;">`Redis`</span> is used as storage  
• <span style="color: #ba372a;">`Rsync`</span> is used to download updates for domains database from SafeDNS

When a user enables a content filtering service, an ISP should set the address of the ISP Go server as the default DNS on the user’s computers or devices (via DHCP, connection scripts, and so on) or forcefully redirect all DNS requests from the such user to the ISP Go server.

Using the ISP-Go API (special HTTP requests executed from the provider's scripts) this DNS server is informed about which categories of sites should not be shown to the user. The API also supports individual custom black and white lists and their global variants, which are valid for all users.

If a user types in the browser address of a prohibited site, the filtering DNS server will respond to a DNS packet from the IP address of the block page, then the browser will load the block page. The block page shows a reason why access to this domain is blocked. The block page design is customizable.

##### Minimal system requirements:

The server on which the ISP-Go service can be deployed and tested with up to 100 users must meet the following minimal requirements:

- Architecture x86-64
- Debian 8, 9 or 10 amd64 installed
- 2GB RAM
- 1.6 GHz 1-core CPU
- 20 GB of free disk space
- 100 Mbit\\s network card

##### Recommended system requirements:

Average usage of the ISP-Go filtering service covers a few thouthands requests per second. To ensure this performance should meet recommended system requirements.

- Architecture x86-64
- Debian 8, 9 or 10 amd64 installed
- 2GB RAM
- 2 GHz 4-cores CPU
- 80 GB of free disk space
- 1 Gbit\\s network card

##### High performance system requirements:

High performance usage of the ISP-Go filtering service covers ~0.5 million requests per second or more. To ensure this performance should meet recommended system requirements.

- Architecture x86-64
- Debian 8, 9 or 10 amd64 installed
- 8 or more GB RAM
- 2 GHz 8-cores CPU or more
- 160 GB or more of free disk space
- 2.5 Gbit\\s network card or two 1 Gbit\\s network cards

Currently, ISP Go is distributed as a deb-package for amd64 architecture.

For the functioning of the filtering server, an ISP should already have a standard recursive caching DNS server. **Bind9** or **Unbound** with the default settings on the other server is well suited, or you can put one of these DNS servers on the same server as ISP Go and configure it to listen only to the address 127.0.0.1.

The filtering DNS server (<span style="color: #ba372a;">`isp-go-dnsproxy`</span>) transfers all non-blocked queries to the caching server and does not cache any data itself. It is possible to pass queries of all ISP’s users (even those for which the filtering service is not enabled) through the filtering DNS server without filtering or with filtering of some default categories.

The "Block page" and "ISP-Go API" web applications are designed as separate daemons, each of them listens to its own port at 127.0.0.1. To transfer requests from the external networks to these web applications, **Nginx** is used, **Nginx** listens to port 80 on the external network interface. Distribution of requests across these web applications is based on the header "Host:" in HTTP requests. All requests with a dedicated hostname for ISP Go API are transferred to the ISP Go API, while all other requests are transferred to the block page.

  
A provider should have any entity (billing, authentication system), which knows the correspondence between internal IP addresses and end users. A prerequisite for the implementation of the ISP Go solution for ISP is the possibility in the existing ISP’s systems to run a script when an IP address is issued to a user and (preferably) when the user logs off. Also, ISP should have a specialist who can write such scripts that will invoke SafeDNS ISP Go API on these events (user’s login and log off).

  
A server on which <span style="color: #ba372a;">`isp-go-dnsproxy`</span> is installed should be able to send HTTP POST requests to licensing control server on the address www.safedns.com. The absence of such requests is a violation of the license.

<p class="callout warning">  
ATTENTION!!! `isp-go-dnsproxy` always rejects non-recursive DNS queries. This is done to eliminate the possibility of an attack like “DNS loop” when isp-go-dnsproxy and Bind transmit requests from one to another without end. Since Bind and other DNS servers always generate non-recursive DNS queries, browsers, and other DNS clients generate only recursive ones, then the loop will be broken. This, however, means that the configuration like “Bind with multiple zones on 127.0.0.1, isp-go-dnsproxy on the external address, and this external address is in NS records in these zones” is unworkable and strongly discouraged. It is not a bug but deliberately introduced limited functionality.</p>

# 2. Before installation

<p class="callout info">The ISP-Go server can be installed on Debian Jessie/Stretch/Buster, choose the amd64 packet. 32bit computers are not supported, arm64, i386, and other packets are not supported.</p>

<p class="callout info">Please make sure that the standard repositories are added to the /etc/apt/sources.list, as this will be required for installing additional ISP Go components (nginx, redis, etc.).</p>

After you installed the Debian amd64 OS please run the following commands. All commands must be executed by the <span style="color: #ba372a;">`root`</span> user.

- Install `<span style="color: #ba372a;">SSH</span>` and <span style="color: #ba372a;">`iptables` </span>if they are not installed yet

```shell
apt-get install ssh iptables
```

- Using `<span style="color: #ba372a;">iptables</span>`, block access to port 53 (both UDP and TCP) from networks that do not belong to the provider. A minimal example for an ISP with a *192.168.5.0/24* network:

```
iptables -A INPUT -p udp --dport 53 ! --src 192.168.5.0/24 -j DROP 
iptables -A INPUT -p tcp --dport 53 ! --src 192.168.5.0/24 -j DROP
```

<p class="callout info">It is impossible to automate this step, because the provider may have some ready-made script for configuring the firewall, and it is a bad idea to add rules blindly on top of it.</p>

- You need to make `<span style="color: #ba372a;">iptables</span>` rules automatically apply after a server reboot. One way to achieve this is to install the package `<span style="color: #ba372a;">iptables-persistent</span>`:

```shell
apt-get install iptables-persistent
```

- Save the rules:

```shell
service netfilter-persistent save
```

# 3. Installation

**Installation Script**

To install **ISP Go**, save, grant execution rights, and then run the script below.

Or execute the commands given in the script in the same sequence in the command line manually.

```shell
#!/bin/bash

LOGIN=$1

PASS=$2

wget https://$LOGIN:$PASS@mirror.safedns.com/repo/ispgo/isp-go-frontend_0.11_all.deb https://$LOGIN:$PASS@mirror.safedns.com/repo/ispgo/isp-go_1.2.1_amd64.deb

apt update

apt install -y curl daemon dbconfig-common rsync redis-server nginx postgresql nodejs

dpkg -i isp-go_1.2.1_amd64.deb isp-go-frontend_0.11_all.deb
```

**Creating an installation script**

Open a text editor that is convenient for you (Nano will be used in the example):

```shell
nano install.sh
```

Copy the commands from the example above into it and save it.

Grant the right of execution for the created script, and then run it using the username and password provided to you by the manager.

```shell
chmod +x install.sh
./install.sh login password
```

In the process of installation the following configuration files will be created:

- `<span style="color: #ba372a;">/etc/isp-go/config.ini</span>` – components configuration
- <span style="color: #ba372a;">`/etc/nginx/conf.d/log_format.conf` </span>– configuration of the block page logs format
- `<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-api</span>` – virtual host for handling API requests
- `<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-blocked</span>` – blockpage virtual host

All files should be manually configured to finish the ISP-Go installation.

# 4. Configuration

#### **Redis configuration**

Set the maximum size of memory allocated for data storage and policy of work when this limit will be achieved in `<span style="color: #ba372a;">/etc/redis/redis.conf</span>` the file. To prevent loss of data you should use these settings:

```shell
maxmemory 2GB 
maxmemory-policy noeviction
```

To apply the settings restart **Redis server** service:

```shell
service redis-server restart
```


#### **ISP Go modules configuration**

All the modules read the same configuration file `<span style="color: #ba372a;">/etc/isp-go/config.ini</span>`  
The file consists of several sections.

##### **\[dnscache\] section**

<details id="bkmrk-this-section-is-used"><summary>\[dnscache\] section</summary>

This section is used by `<span style="color: #ba372a;">isp-go-dnsproxy</span>` and contains only one key - forward. As a value, you should set an IP address and a port of caching DNS server which will be used to resolve all non-blocked DNS requests.

Example:

```shell
[dnscache] 
forward = 8.8.8.8:53
```


</details>**\[proxy\] section**

<details id="bkmrk-this-section-is-used-0"><summary>\[proxy\] section</summary>

This section is used by `<span style="color: #ba372a;">isp-go-dnsproxy</span>` and contains listen ***blockpage-ip**, **log**,* and ***PID*** keys.  
The ‘Listen’ key contains an IP address and a port on which `<span style="color: #ba372a;">isp-go-dnsproxy</span>` should get DNS requests from users. It can be duplicated to accept requests on several addresses at once, for example, IPv4 and IPv6 at the same time. The number of keys, as well as the number of listening addresses, is unlimited

`<span style="color: #ba372a;">blockpage-ip</span>` key contains an IP address of a block page, which should be passed to users in answers to blocked requests. This is the IP address where **Nginx** accepts requests to the block page. In general, case when filtering DNS server and **Nginx** are on the same server you should set for this key the same IP address as for the listening key. The key can be duplicated to specify an alternative address for the block pages, for example, for IPv6. It is allowed to use at least one key only for A redirects or only AAAA packets, but no more than two to support both IP versions.

<p class="callout warning">`<span style="color: #ba372a;">log, pid</span>` keys contain the absolute path to the log file and PID file accordingly. To prevent loss of compatibility with init scripts from the ISP Go package you should not change the path to the PID file.</p>

Example:

```shell
[proxy] 
listen = 192.168.5.1:53 ; IPv4 
listen = [4321:a:bcde:1::2020]:53 ; IPv6 
blockpage-ip = 192.168.5.1 ; IPv4 to forwarding A 
blockpage-ip = abcd:1234:zyxw::9876 ; IPv6 to forwarding AAAA 
log = /var/log/isp-go/isp-go-dnsproxy.log 
pid = /var/log/isp-go/isp-go-dnsproxy.pid
```


</details>**\[datafiles\] section**

<details id="bkmrk-this-section-contain"><summary>\[datafiles\] section</summary>

This section contains '**path', 'file', 'cats'** keys. **‘file’** key can be used several times in this section.

The **‘path’** key contains the absolute path to the directory with the SafeDNS domain database. Files, that database is made of, should be enumerated in the **‘file’** keys. The order of these keys is important for correct work. Each top-listed file is processed as a correction to all bottom-listed files. To maintain the correct work of the database we prohibit the change of default values for these keys.

  
The **‘cats’** key contains the absolute path to the JSON file with the list of supported categories. You cannot change this file, because all changes made to it, will be lost on the ISP Go package update. If you need to hide some categories or translate them to some other language you should create a copy of the **catgroups.json** file and make all changes in the copy. The numbers of the categories should not be changed because they are linked to the content of the master database of SafeDNS.

Example:

```shell
[datafiles] 
path = /var/lib/isp-go/filter/ 
file = host2cat-fast.dat file = host2cat.dat 
cats = /usr/share/isp-go/config/
```


</details>**\[blockpage\] section**

<details id="bkmrk-this-section-is-used-1"><summary>\[blockpage\] section</summary>

This section is used by the `<span style="color: #ba372a;">isp-go-blockpage</span>` application and contains '**listen'**, '**templates'**, '**log'**, and '**pid'** keys.  
**‘listen’** key contains an IP address (usually 127.0.0.1) and a port that `<span style="color: #ba372a;">isp-go-blockpage</span>` daemon listens on for HTTP requests to block page.

<p class="callout warning">Daemon isp-go-blockpage does not accept requests from users. All requests should be passed through **Nginx**.</p>

The **‘templates’** key contains the absolute path to the directory with templates of block page. Do not change templates installed with the package, because all changes made will be lost on the ISP Go package update. We recommend copying the whole directory <span style="color: #ba372a;">`/usr/share/isp-go/templates`</span> and changing templates in this copy.

**‘log’** and **‘pid’** keys contain the absolute path to log and PID files accordingly. To prevent loss of compatibility with init scripts from the ISP Go package you should not change the path to pidfile.

Example:

```shell
[blockpage] 
listen = 127.0.0.1:8081 
templates = /usr/share/isp-go/templates/ 
log = /var/log/isp-go/isp-go-blockpage.log 
pid = /var/run/isp-go/isp-go-blockpage.pid
```


</details>**\[api\] section**

<details id="bkmrk-this-section-is-used-2"><summary>\[api\] section</summary>

This section is used by the web application `<span style="color: #ba372a;">isp-go-api</span>` and contains '**listen'**, '**log'**, and '**pid'** keys.  
**‘listen’** key contains an IP address (usually 127.0.0.1) and a port that `<span style="color: #ba372a;">isp-go-api</span>` daemon listens on for HTTP requests to API. An IP address or a port should be different from the set in the **\[blockpage\]** section

<p class="callout warning">Listening on an externally accessible IP address will be a security problem. The isp-go-api daemon does not contain any authorization mechanisms, so anyone who can send a request can make any changes to user settings (including someone else's). To prevent such a situation, it is recommended to use the IP address 127.0.0.1 here and to implement external access (with authorization) at the Nginx level.</p>

Example:

```shell
[api] 
listen = 127.0.0.1:8080 
log = /var/log/isp-go/isp-go-api.log 
pid = /var/run/isp-go/isp-go-api.pid
```


</details>##### **\[common\] section**

<details id="bkmrk-%5Bcommon%5D-section-the"><summary>\[common\] section</summary>

The section is used by all three daemons and contains the keys '**redis-ip'** and '**redis-port'**.

The **'redis-ip'** and **'redis-port'** keys specify which **Redis** server the daemons that are part of ISP Go should connect to. For performance reasons, it is recommended to run the **Redis** server on the same machine where ISP Go is installed.

Example:

```shell
[common] 
redis-ip = 127.0.0.1 
redis-port = 6379
```

</details>**Enabling ISP Go services**

To enable ISP Go services automatic startup please run the following commands:

```
systemctl enable isp-go-dnsproxy
systemctl enable isp-go-blockpage
systemctl enable isp-go-api
```

**Applying settings**

After changing the configuration file you should restart all ISP Go services:

```shell
service isp-go-dnsproxy restart 
service isp-go-blockpage restart 
service isp-go-api restart
```

#### **Nginx configuration**

**Nginx** is used in ISP Go for the following tasks:  
• separation of API requests from requests to the block page and to the administrative web interface  
• proxying requests to corresponding web applications  
• restricting access to the API and to the administrative web interface

For correct separation of requests, you need to register the domain name used to manage filtering through the API in the `<span style="color: #ba372a;">server_name</span>` directive of the file `<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-api</span>` instead of the value 'api.ispgo'.

All other requests will go to the virtual host configured in the file `<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-blocked</span>` due to the presence of the `<span style="color: #ba372a;">default_server</span>` modifier in the <span style="color: #ba372a;">`listen`</span> directive.

You can create additional virtual hosts with the following exceptions:  
• each additional virtual host should contain the `<span style="color: #ba372a;">server_name</span>` directive  
• `<span style="color: #ba372a;">default_server</span>` option cannot be used  
• HTTPS usage is not recommended, because users who are requesting blocked websites via HTTPS will get browser warnings of an invalid SSL certificate.

IP addresses and ports in the `<span style="color: #ba372a;">proxy_pass</span>` directive in **Nginx** configuration files should correspond with IP addresses and ports on which web applications were launched (see **listen** key in`<span style="color: #ba372a;"> /etc/isp-go/config.ini</span>` file).

API access is restricted using the `<span style="color: #ba372a;">allow</span>` and `<span style="color: #ba372a;">deny</span>` directives. Directives are processed in turn from top to bottom until the first match. The default configuration allows access only from the address **127.0.0.1**. You must allow access from the server where the billing system is installed.

<p class="callout danger">In no case, access should be allowed to the API from untrusted (including user) systems, because if access to the API is provided an attacker can change any filtering settings for any users.</p>

To apply changes reload **Nginx**:

```shell
service nginx reload
```

#### **Database update**

In the installation process demo version of the database will be copied to `<span style="color: #ba372a;">/var/lib/isp-go/filter/</span>` folder. For production deployment, you should replace the demo version with the full one and configure automatic updates.  
The domain database is updated by **cron** using **rsync**. You need an **ssh key** to authorize access to the **safedns.com** server. To get auto-update to perform the following steps:

Generate an ssh key that will be used to download domain database updates:

```shell
mkdir safedns-key 
cd safedns-key 
ssh-keygen -t rsa -N "" -f id_rsa
```

As a result, the files `<span style="color: #ba372a;">id_rsa</span>` (private key, which must be kept strictly secret and not lost) and `<span style="color: #ba372a;">id_rsa.pub</span>` (public key) will be created.

Send the created `<span style="color: #ba372a;">id_rsa.pub</span>` file to the technical support team at <support@safedns.com> Don't need to send `<span style="color: #ba372a;">id_rsa</span>` antwhere!

The technical support team will notify you when the ssh key will be authorized on the SafeDNS server.

Copy the `<span style="color: #ba372a;">id_rsa</span>` and `<span style="color: #ba372a;">id_rsa.pub</span>` files to the directory where the update script is looking for them:

```shell
mkdir -p -m 0755 /var/lib/isp-go/.ssh 
cd safedns-key 
cp id_rsa id_rsa.pub /var/lib/isp-go/.ssh/ 
chown -R isp-go:isp-go /var/lib/isp-go/.ssh
```

Wait for 1 hour and make sure that `<span style="color: #ba372a;">host2cat.dat</span>` and <span style="color: #ba372a;">`host2cat-fast.dat`</span> files in the directory `<span style="color: #ba372a;">/var/lib/isp-go/filter</span>` have been updated.

You can also update the data-files manually, just run the following commands:

```shell
su isp-go -c 'rsync -rtv --progress safedns-isp@safedns.com:host2cat.dat ~/filter/' 
su isp-go -c 'rsync -rtv --progress safedns-isp@safedns.com:host2cat-fast.dat ~/filter/'
```

<p class="callout info">The server should have access to www.safedns.com on TCP port 443.</p>

#### **Setting up sending the statistics**

To do this, you need to allow outgoing connections from the isp-go server to [www.safedns.com](https://www.safedns.com) on **TCP** port **443** and perform all the steps from Setting up automatic updating of the domain database (previous part).

To check the correctness of sending statistics on the isp-go server, please perform:

```shell
curl -f -X POST --data "key=`cut -d ' ' -f 2 /var/lib/isp-go/.ssh/id_rsa.pub | base64 -d | md5sum | cut -d ' ' -f1`&count=`redis-cli --raw hlen ip`" https://www.safedns.com/isp-kit-dog/
```

**OK** is the correct answer to this command.

#### **Block page design**

By default, ISP Go comes with a minimal, strict, and ascetic design of the lock page. To change this design, you need to edit the **HTML** templates that are located at `<span style="color: #ba372a;">/usr/share/isp-go/templates/</span>`<span style="color: #ba372a;"><span style="color: #000000;">, where</span></span> `<span style="background-color: #ffffff; color: #ba372a;">base.html</span>` – is the main template file, and the rest are inherited from it. The syntax of templates is described in the Go language guide:

• [https://golang.org/pkg/text/template/](https://golang.org/pkg/text/template/)  
• [https://golang.org/pkg/html/template/](https://golang.org/pkg/html/template/)

The following variables are available:

• Domain: requested hostname in the Host field of HTTP request header  
• Cats: array with category names of the blocked websites. This variable is allowed to use with the `<span style="color: #ba372a;">blocked_by_category.html</span>` template file only.

To add images to a block page we recommend setting a separate virtual host for image storing and using an absolute **URL** to an image with `<span style="color: #ba372a;"><img></span>` tag (example: `<span style="color: #ba372a;"><img>http://img.isp.com/block-img.png</img></span>`)

To apply changes restart the `<span style="color: #ba372a;">isp-go-blockpage</span>` service:

```shell
service isp-go-blockpage restart
```

#### **Peculiarities of listening on specific IP addresses**

It is possible to use the **0.0.0.0** IP address in the listen key of a `<span style="color: #ba372a;">[proxy]</span>` section of the configuration file if you use caching DNS server installed on another server. In this case, `<span style="color: #ba372a;">isp-go-dnsproxy</span>` will process requests on all network interfaces.  
If you have caching DNS server installed on the same server (and listening on **127.0.0.1** IP address), you should designate a specific IP address from one of the network interfaces.

#### **User activity statistics**

Statistics are recorded to a **CSV** file and imported into the **PostgreSQL** database every 5 minutes. Statistics are stored in the database in "raw" and aggregated form. "Raw" statistics are stored for 1 day. Aggregated statistics are stored for 3 months.  
The statistics are recalculated every five minutes.

If a user has more than one IP address, then statistics for all addresses are summarized. Likewise for anonymous users.  
The following reports are available:  
• number of requests by hours and days for the period;  
• number of requests to the top 100 domains;  
• detailed statistics by domains and days;  
• "Raw" statistics for the last hour.  
Statistics are accessed through requests to the **ISP Go REST API.**

# 5. Examples of API requests

After the initial installation, you need to write a script that will change user settings. The settings are changed using **HTTP** requests to the **ISP Go API**. The examples below use **curl** from the command line to generate requests.

```shell
# Adding and removing user IP
curl -X PUT -d "[\"192.168.5.1\"]" http://api.isp.com/users/aep/ip/ 
curl http://api.isp.com/users/aep/ip/ 
curl -X DELETE http://api.isp.com/users/aep/ip/ 
curl http://api.isp.com/users/aep/ip/ 
curl -X PUT -d "[\"192.168.5.1\"]" http://api.isp.com/users/aep/ip/ 

# Categories list in json format. 
curl http://api.isp.com/categories/ 
curl http://api.isp.com/categorygroups/ 

# Global white list # Replacing the entire list 
curl -X PUT -d '["magazine.com", "feed.com"]' http://api.isp.com/whitelist/ 
curl http://api.isp.com/whitelist/ 

# Add and remove one record at a time 
curl -X POST -d '["magazine.com"]' http://api.isp.com/whitelist/ 
curl -X DELETE http://api.isp.com/whitelist/feed.com 
curl http://api.isp.com/whitelist/ 

# Global black list works similarly 
curl http://api.isp.com/blacklist/ 

# Filter configuring:
# Complete replacement of the list of blocked categories: 
curl -X PUT -d '[3, 4, 5, 11]' http://api.isp.com/users/aep/filter/ 

# Add and remove one category at a time: 
curl -X POST -d '[6]' http://api.isp.com/users/aep/filter/ 
curl -X DELETE http://api.isp.com/users/aep/filter/5 
curl http://api.isp.com/users/aep/filter/ 

# black and white lists of users: 
# complete replacement:
curl -X PUT -d '["dark.com", "orange.com", "red.com", "cyan.com"]' http://api.isp.com/users/aep/blacklist/ 

# Add and remove one domain at a time: 
curl -X POST -d '["antigreen.com"]' http://api.isp.com/users/aep/blacklist/ 
curl -X DELETE http://api.isp.com/users/aep/blacklist/orange.com 
curl http://api.isp.com/users/aep/blacklist/ 

# White list works similarly 
curl http://api.isp.com/users/aep/whitelist/
```

To enable the mode "Use the white list only" for a user you need to add a root domain to the black list:

```shell
curl -X POST -d '["-"]' http://api.isp.com/users/aep/blacklist/
```

To remove the root domain, use:

```shell
curl -X DELETE http://api.isp.com/users/aep/blacklist/-
```

<p class="callout warning">There is a non-removable and non-editable whitelist with the highest priority, which is not accessible through any administration interface, which includes safedns.com addresses, antivirus update sites, Windows update sites, etc.</p>

<details id="bkmrk-an-example-of-a-php-"><summary>An example of a PHP script for accessing the API</summary>

```PHP
<?php
// The examples file requires PHP curl extension installed 
// Address of the API server 
$server = 'http://api.isp.com'; 
function api_request($url, $method = 'GET', $data = null) { 
	$ch = curl_init($url); 
	curl_setopt ($ch, CURLOPT_HEADER, 0); 
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
	curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method); 
	if (!is_null($data)) curl_setopt($ch, CURLOPT_POSTFIELDS, $data); 
	$out = curl_exec($ch); 
  
  	$return_code = curl_getinfo($ch, CURLINFO_HTTP_CODE); 
  
  	switch ($return_code) { 
      case 200: 
        if (strlen($out)) return json_decode($out); 
        else return "OKrn"; 
        break; 
      case 400: 
        throw new Exception('Bad request'); 
        break; 
      default: throw new Exception('Error');
    } 
} 
/* 
User record is created automatically when trying 
To record any information about the user
*/ 

// Add to a user with an identifier aep ip-address 192.168.5.1 
print api_request($server."/users/aep/ip/", 'POST', '["192.168.5.1"]');

// Replace the list of ip-addresses of the user aep to 192.168.5.1, 192.168.5.2 
print api_request($server."/users/aep/ip/", 'PUT', '["192.168.5.1", "192.168.5.2"]'); 

// Delete ip-addresses 192.168.5.1 from the list of addresses of the user aep 
print api_request($server."/users/aep/ip/192.168.5.1", 'DELETE'); 

// Return a list of ip-addresses for the user aep 
$result = api_request($server."/users/aep/ip/", 'GET'); 
var_dump($result); 

// Delete the whole list of ip-addresses of the user aep 
print api_request($server."/users/aep/ip/", 'DELETE'); 
?>

```

</details>

# 6. Backup and restore

Set periodic backup of directories:  
`<span style="color: #ba372a;">/etc/isp-go</span>`  
`<span style="color: #ba372a;">/var/lib/isp-go/.ssh</span>`

Set periodic backup of **Redis** database (you can get file name and path from the dbfilename and dir parameters of **/etc/redis/redis.conf** configuration file).

Restore process:

• Install **ISP-Go**. During the installation process, specify **127.0.0.1** as the IP from which access to the API is allowed in the `<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-api</span>` file. Do not make API requests until recovery is complete;  
• Restore **Redis** database:  
 1. Stop Redis service on the new instance - sudo service redis-server stop  
 2. Copy the dump.rdb file to /var/lib/redis/ on the new instance  
 3. Start Redis service on the new instance - sudo service redis-server start  
• Restore files to `<span style="color: #ba372a;">/etc/isp-go</span>` and `<span style="color: #ba372a;">/var/lib/isp-go/.ssh</span>` from backup directories;  
• Restart `<span style="color: #ba372a;">redis-server</span>`, `<span style="color: #ba372a;">isp-go-dnsproxy</span>`, `<span style="color: #ba372a;">isp-go-api</span>` and `<span style="color: #ba372a;">isp-go-blockpage</span>`<span style="color: #ba372a;"> <span style="color: #000000;">;</span></span>

# 7. ISP-Go slave server

Install second (slave) **ISP-Go** server. Make sure that `<span style="color: #ba372a;">isp-go-api</span>` is launched on the master server only. All requests to the API should be sent to the master server.

  
To prevent `<span style="color: #ba372a;">isp-go-api</span>` launch on the slave server and block requests proxying, execute commands:

```shell
service isp-go-api stop 
update-rc.d isp-go-api disable 
rm -f /etc/nginx/sites-enabled/isp-go-api 
service nginx restart
```

Configure replication between **Redis** instances. To do this, you need to allow the main server to listen at network addresses other than **127.0.0.1** by setting the `<span style="color: #ba372a;">bind</span>` parameter in `<span style="color: #ba372a;">/etc/redis/redis.conf</span>` to **0.0.0.0**.

```JSON
bind 0.0.0.0
```

or, if needed, you can set it to an IP or IPs of the server

```JSON
bind 127.0.0.1 192.168.5.100 10.0.0.100
```

To apply new settings restart **Redis** on the master server:

```shell
service redis-server restart
```

<p class="callout danger">You should restrict access to port **TCP/6379** on the master server which is used by **Redis** to listen. The port should be closed using **iptables** and stay accessible only for the loopback interface (needed for `<span style="color: #ba372a;">isp-go-dnsproxy</span>` and `<span style="color: #ba372a;">isp-go-blockpage</span>`) and the slave server. An attacker, having access to the **Redis** server via **TCP**, could change any setting of any user, or even worse, force **Redis** to take up all available memory.</p>

Consider an example where the master server has an IP address of **192.168.5.100** and the slave server has an IP address of **192.168.5.200**. In this case, security on the main server is ensured by this **iptables** rule:

```shell
iptables -A INPUT ! -s 192.168.5.200 -p tcp --dport 6379 ! -i lo -j DROP
```

To save this rule, so that it recovers after a reboot, run the command:

```shell
service netfilter-persistent save
```

Add to the configuration file `<span style="color: #ba372a;">/etc/redis/redis.conf</span>` on the slave server following parameters at the end of the document:

```Nginx
slaveof <masterip> <masterport>
```

Then restart **Redis** on the slave server to apply the settings:

```shell
service redis-server restart
```

# 8. Migrating from previous versions

There are two utilities for migrating from previous versions:

`<span style="color: #ba372a;">isp-go-dumpdb</span>`

`<span style="color: #ba372a;">isp-go-loaddb</span>`

The first one creates a dump of the user base through the API, and the second one downloads this dump.

Example:

```shell
isp-go-dumpdb -file=./dump.json -server=oldapi.myserver.com 
isp-go-loaddb -file=./dump.json -server=newapi.myserver.com
```

Where **file** is the path to the file where the database dump will be written, and **server** is the name of the host where the API is located.  
API availability can be checked by command:

```shell
curl -v http://api.myserver.com/active_users/
```

If the error **E403 Forbidden** occurs, open access in following the configuration files:

`<span style="color: #ba372a;">/etc/nginx/sites-available/isp-go-api</span>` for ISP-Go

`<span style="color: #ba372a;">/etc/apache2/sites-available/isp.comf</span>` for ISP-Kit

<p class="callout info">Don't forget to restart services after any chages made in configuration files.</p>

# 9. ISP Go API

<details id="bkmrk-users-%5Buser_id%5D-is-t"><summary>Users</summary>

`<span style="color: #ba372a;">[user_id]</span>` is the ISP's user ID. It can contain the characters A-Z, a-z, 0-9 underscore and dash.

The maximum length is 32 characters.

##### Creating a user

A new user is created when trying to create a property for him: specify categories, assign ip.

```Python
POST /users/[user_id]/ip/[“ip_address”]
```

##### Deleting a user

```
DELETE /users/[user_id]
```

##### Check a user existence:

```
HEAD /users/[user_id]
```

If a user is not found returns HTTP response code 404.

##### Getting a list of active users

```
GET /active_users/
```

Response format:

```JSON
["geek", "bugtest", "hammer"]
```

Active users are users which have at least one IP address.

##### Getting a list of all users

```
GET /users
```

Response format:

```JSON
[{
  "name": "geek",
  "safesearch": "off",
  "safeyoutube": "off",
  "status": "enabled",
  "filter": [1, 2],
  "ip": ["192.168.5.6", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"],
  "whitelist": ["facebook.com", "google.com"],
  "blacklist": []
},{
  "name": "bugtest",
  "safesearch": "off",
  "safeyoutube": "on",
  "status": "enabled",
  "filter": [1, 2, 3],
  "ip": ["192.168.5.8"],
  "whitelist": ["reddit.com", "google.com"],
  "blacklist": []
}]
```

By default, returns first 100 users.

```
GET /users?start=100&stop=200
```

Start and stop parameters indicate that should be returned users with `<span style="color: #ba372a;">user_id</span>` from 100 to 200.

##### Search a user

```
GET /search/user1
```

Response format:

```JSON
[{
  "name": "user1",
  "safesearch": "off",
  "safeyoutube": "off",
  "status": "enabled",
  "filter": [],
  "ip": [],
  "whitelist": [],
  "blacklist": []
}]
```

The search is possible by IP or username. Wildcards or masks can be used in the search.

```
GET /search/192.168.0.*
```

Returns all users which IP address starts with `<span style="color: #ba372a;">192.168.0.</span>`

```
GET /search/user*
```

Returns all users which name starts with `<span style="color: #ba372a;">user</span>`

##### GET user’s information

```
GET /users/[user_id]
```

Response format:

```JSON
{
  "name": "[user_id]",
  "safesearch": "off",
  "safeyoutube": "off",
  "status": "enabled",
  "filter": [3, 4, 5, 6],
  "ip": ["192.168.5.6", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"],
  "whitelist": ["reddit.com", "google.com"],
  "blacklist": []
}
```

##### Update user’s information

```JSON
PUT /users/[user_id]

Content-type: application/json

{
  "name": "[user_id]"
  "safesearch": "on",
  "safeyoutube": "off",
  "status": "enabled",
  "filter": [2, 3, 4, 5],
  "ip": ["192.168.5.8", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"],
  "whitelist": ["docs.google.com", "atlassian.net"],
  "blacklist" ["youtube.com", "pornhub.com"]
}
```

##### Disable filtering

This feature allows users to suspend filtering without settings reset.

```JSON
POST /users/[user_id]/status/disabled

Content-type: application/json
```

##### Enable filtering

```JSON
POST /users/[user_id]/status/enabled

Content-type: application/json
```


</details><details id="bkmrk-safesearch-enabling-"><summary>SafeSearch</summary>

##### Enabling safe search by default for a user

```JSON
PUT /config/

Content-type: application/json

{ "safesearch": true }
```

##### Disabling safe search by default for a user

```JSON
PUT /config/

Content-type: application/json

{ "safesearch": false }
```

##### Enabling safe search by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "safesearch": true }
```

##### Disabling safe search by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "safesearch": false }
```

##### Enabling safe Search for `<span style="color: #ba372a;">[user_id]</span>`

```JSON
POST /users/[user_id]/safesearch/on

Content-type: application/json
```

##### Disabling safe Search for `<span style="color: #ba372a;">[user_id]</span>`

```
POST /users/[user_id]/safesearch/off

Content-type: application/json
```

```JSON
PUT /userconfig/

Content-type: application/json

{ "safeyoutube": true }
```


</details><details id="bkmrk-safe-youtube-if-this"><summary>Safe Youtube</summary>

If this option is enabled, the user will be automatically redirected to the safe version of YouTube.

##### Enabling safe YouTube by default

```JSON
PUT /config/

Content-type: application/json

{ "safeyoutube": true }
```

##### Disabling safe YouTube by default

```JSON
PUT /config/

Content-type: application/json

{ "safeyoutube": false }
```

##### Enabling safe YouTube by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "safeyoutube": true }
```

##### Disabling safe YouTube by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "safeyoutube": false }
```

</details><details id="bkmrk-user-addresses-get-t"><summary>User addresses</summary>

##### Get the user’s IP address

```JSON
GET /users/[user_id]/ip/
```

Response format:

```JSON
["192.168.5.6", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"]
```

##### Add user’s IP address (will be added to existing)

```JSON
POST /users/[user_id]/ip/

Content-type: application/json

["127.0.0.1"]
```

or a list of IP addresses:

```JSON
POST /users/[user_id]/ip/

Content-type: application/json

["127.0.0.1", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"]
```

##### Update user’s IP address (existing IP address will be replaced)

```JSON
PUT /users/[user_id]/ip/

Content-type: application/json

["127.0.0.1"]
```

or a list of IP addresses:

```JSON
PUT /users/[user_id]/ip/

Content-type: application/json

["127.0.0.1", "2001:0db8:85a3:0000:0000:8a2e:0370:7334"]
```

##### Delete all user’s IP addresses (disables filtering for the user)

```JSON
DELETE /users/[user_id]/ip/
```

##### Deleting one of the user's IP addresses

```JSON
DELETE /users/[user_id]/ip/[127.0.0.1]
```

</details><details id="bkmrk-general-information-"><summary>General information</summary>

##### Get the grouped list of categories with the names and identifiers of the categories

In order to get categories in the language other than the default one, you should add the HTTP header Accept-Language to the request with language indication (for example pt\_BR).

```JSON
GET /categorygroups/
```

Response format:

```JSON
[{
  "group": "Security",
  "categories": {
    "3": "Virus Propagation",
    "4": "Phishing",
    ...
  }
}, {
  "group": "Illegal Activity",
  "categories": {
    "6" : "Drugs",
    ...
  }
},
...
]
```

##### Get a list of categories with names and identifiers of categories

```
GET /categories/
```

Response format:

```JSON
{
  "3":"Virus Propagation",
  "4":"Phishing", 
  …
}
```

##### Check a website

```JSON
GET /site/facebook.com
```

Response format:

```JSON
{
  "domain": "facebook.com",
  "categories": [29],
}
```

</details><details id="bkmrk-categories-for-block"><summary>Categories for blocking</summary>

##### Adding a category to the list of blocked by default

```JSON
PUT /config/

Content-type: application/json

{ "filter": [1, 2] }
```

##### Removing a category from the blocked by default

```JSON
PUT /config/

Content-type: application/json

{ "filter": [] }
```

##### Adding a category to the list of blocked by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "filter": [1, 2] }
```

##### Removing a category to the list of blocked by default for users

```JSON
PUT /userconfig/

Content-type: application/json

{ "filter": [] }
```

##### Get a list of user’s categories

```
GET /users/[user_id]/filter/
```

Response format:

```JSON
[1, 2, 3]
```

##### Add category to user’s list

```JSON
POST /users/[user_id]/filter/

Content-type: application/json

[1, 2]
```

##### Save new user’s list of categories

```JSON
PUT /users/[user_id]/filter/

Content-type: application/json

[1, 2, 3]
```

##### Disable one category for a user

```JSON
DELETE /users/[user_id]/filter/2
```

##### Disable all categories for a user

```JSON
DELETE /users/[user_id]/filter/
```

</details><details id="bkmrk-black-list-get-the-u"><summary>Black list</summary>

##### Get the user’s black list

```JSON
GET /users/[user_id]/blacklist/
```

Response format:

```JSON
["google.com", "facebook.com"]
```

##### Add a domain to the user’s black list

```JavaScript
POST /users/[user_id]/blacklist/

Content-type: application/json

["www.facebook.com"]
```

##### Replace to new black list

```JSON
PUT /users/[user_id]/blacklist/

Content-type: application/json

["www.facebook.com"]
```

##### Remove a domain from the user’s black list

```JSON
DELETE /users/[user_id]/blacklist/facebook.com
```

##### Delete the user’s black list

```
DELETE /users/[user_id]/blacklist/
```

##### Setting the "White list only" option

For the user to work in the "everything is prohibited except what is explicitly whitelisted" mode, you need to blacklist the root domain:

```JSON
POST /users/[user_id]/blacklist/

Content-type: application/json

["-"]
```

To delete the root domain from the blacklist:

```JSON
DELETE /users/[user_id]/blacklist/-
```

</details><details id="bkmrk-white-list-get-the-u"><summary>White list</summary>

##### Get the user’s white list

```JSON
GET /user/[user_id]/whitelist/
```

Response format:

```JSON
["google.com"]
```

##### Add domain to a user’s white list

```JSON
POST /users/[user_id]/whitelist/

Content-type: application/json

["google.com"]
```

##### Replace the user’s white list

```JSON
PUT /users/[user_id]/whitelist/

Content-type: application/json

["google.com"]
```

##### Remove a domain from the user’s white list

```JSON
DELETE /users/[user_id]/whitelist/google.com
```

##### Delete the user’s white list

```
DELETE /users/[user_id]/whitelist
```

</details><details id="bkmrk-global-blacklist-%28ta"><summary>Global blacklist (takes precedence over user lists)</summary>

##### Get the global black list

```
GET /blacklist
```

Response format:

```JSON
["google.com"]
```

##### Add domain to global black list

```JSON
POST /blacklist/

Content-type: application/json

["www.google.com"]
```

##### Replace to new global black list

```JSON
PUT /blacklist/

Content-type: application/json

["www.google.com"]
```

##### Remove a domain from the global black list

```JSON
DELETE /blacklist/google.com
```

##### Delete the global black list

```
DELETE /blacklist/
```

Setting the option "The global whitelist only"

For the user to work in the "everything is prohibited except what is explicitly whitelisted" mode, you need to blacklist the root domain:

```JSON
POST /blacklist/

Content-type: application/json

["-"]
```

To delete the root domain from the blacklist:

```JSON
DELETE /blacklist/-
```


</details><details id="bkmrk-global-white-list-ge"><summary>Global white list</summary>

##### Get the global white list

```
GET /whitelist/
```

Response format:

```JSON
["google.com"]
```

##### Add a domain to the global white list

```JSON
POST /blacklist/

Content-type: application/json

["www.google.com"]
```

##### Replace to new global white list

```JSON
PUT /blacklist/

Content-type: application/json

["www.google.com"]
```

##### Remove a domain from the global white list

```JSON
DELETE /whitelist/google.com
```

##### Delete the global white list

```
DELETE /whitelist/
```

</details><details id="bkmrk-reports-user-activit"><summary>Reports</summary>

##### User activity by hour

Where `<span style="color: #ba372a;">[date_from]</span>`, `<span style="color: #ba372a;">[date_to]</span>` are dates in **YYYY-MM-DD** format.

```JSON
GET /users/[user_id]/stat/activity/hour?from=[date_from]&to=[date_to]

{
  "labels": ["2022-06-29 14:00:00", "2022-06-29 15:00:00"],
  "datasets": [{
    "label": "Requests",
    "data": [375, 275]
  },{
    "label": "Blocks",
    "data": [13, 0]
  }]
}
```

Where:  
`<span style="color: #ba372a;">labels</span>` - a list of timestamps,  
`<span style="color: #ba372a;">datasets</span>` - a list of objects containing data for charts and legends,  
`<span style="color: #ba372a;">data</span>` - a list of values corresponding to timestamps,  
`<span style="color: #ba372a;">label</span>` - chart name (Requests - number of requests, Blocks - number of blocks).

##### User activity by day

Where `<span style="color: #ba372a;">[date_from]</span>`, `<span style="color: #ba372a;">[date_to]</span>` are dates in **YYYY-MM-DD** format.

```JSON
GET /users/[user_id]/stat/activity/day?from=[date_from]&to=[date_to]

{
  "labels": ["2022-06-29", "2022-06-30"],
  "datasets": [{
    "label": "Requests",
    "data": [5770, 3456]
  },{
    "label": "Blocks",
    "data": [8, 9]
  }]
}
```

Where:  
`<span style="color: #ba372a;">labels</span>` - a list of timestamps,  
`<span style="color: #ba372a;">datasets</span>` - a list of objects containing data for charts and legends,  
`<span style="color: #ba372a;">data</span>` - a list of values corresponding to timestamps,  
`<span style="color: #ba372a;">label</span>` - chart name (Requests - number of requests, Blocks - number of blocks).

##### Domains

```JSON
GET /users/[user_id]/stat/domains/[filter]?from=[date_from]&to=[date_to]

{
  "labels": ["example.com", "google.com", "asdfg.com"],
  "datasets": [{
    "label": "Requests",
    "data": [630, 474, 290]
  },{
    "label": "NXdomain",
    "data": [0, 0, 290]
  },{
    "label": "Blocks",
    "data": [630, 0, 0]
  }]
}
```

Where:  
`<span style="color: #ba372a;">[date_from]</span>`, `<span style="color: #ba372a;">[date_to]</span>` - dates in **YYYY-MM-DD** format,  
`<span style="color: #ba372a;">[filter]</span>` - can be set to all, www  
`<span style="color: #ba372a;">all</span>` - returns all domains,  
`<span style="color: #ba372a;">www</span>` - returns only domains start with www  
`<span style="color: #ba372a;">labels</span>` - a list of domains,  
`<span style="color: #ba372a;">datasets</span>` - a list of objects containing data for charts and legends,  
`<span style="color: #ba372a;">data</span>` - a list of values corresponding to timestamps,  
`<span style="color: #ba372a;">label</span>` - chart name (**Requests** - number of requests, **Blocks** - number of blocks, **NXdomain** - number of requests to non-existing domains).

##### Detailed

```JSON
GET /users/[user_id]/stat/detail?from=[date_from]&to=[date_to]

{
  "timestamps": ["2022-06-29 15:00:00", "2022-06-29 16:00:00"],
  "reports": [{
    "labels": ["clients4.google.com", "www.google.ru"],
    "cats": [[48, 251], [48]],
    "datasets": [{
      "label": "Requests",
      "data": [29, 20]
    },{
      "label": "NXdomain",
      "data": [0, 0]
    },{
      "label": "Blocks",
      "data": [0, 0]
    }]
  },{
    "labels": ["live.github.com", "lb._dns-sd._udp.0.0.200.10.in-addr.arpa", "ssl.gstatic.com"],
    "cats": [[2], [2], [2]],
    "datasets": [{
      "label": "Requests",
      "data": [73, 48, 48]
    },{
      "label": "NXdomain",
      "data": [0, 48, 0]
    },{
      "label": "Blocks",
      "data": [0, 0, 0]
    }]
  }]
}
```

Where:  
`<span style="color: #ba372a;">[date_from]</span>`, `<span style="color: #ba372a;">[date_to]</span>` - dates in **YYYY-MM-DD** format,  
`<span style="color: #ba372a;">timestamps</span>` - a list of timestamps,  
`<span style="color: #ba372a;">reports</span>` - a list of appropriate intervals reports,  
`<span style="color: #ba372a;">labels</span>` - a list of domains,  
`<span style="color: #ba372a;">cats</span>` - lists of categories for domains,  
`<span style="color: #ba372a;">datasets</span>` - a list of objects containing data for charts and legends,  
`<span style="color: #ba372a;">data</span>` - a list of values associated with each domain,  
`<span style="color: #ba372a;">label</span>` - chart name (**Requests** - number of requests, **Blocks** - number of blocks, **NXdomain** - number of requests to non-existing domains).

##### Raw statistics

Full user statistics for the last hour. Updates every 5 minutes.

```JSON
GET /users/[user_id]/stat/raw

{
  "fields": ["created", "nxdomain", "address", "host", "blockedhost", "reasom", "cats", "blockedcats"],
  "data": [
    ["2016-07-11 17:50:26", "0", "10.200.1.88", "tpc.googlesyndication.com", "tpc.googlesyndication.com", "4", [2], [0]],
    ["2016-07-11 17:50:26", "0", "10.200.1.88", "tpc.googlesyndication.com", "tpc.googlesyndication.com", "4", [2], [0]]
  ]
}
```

</details><details id="bkmrk-response-codes-200-o"><summary>Response codes</summary>

`<span style="color: #ba372a;">200</span>` OK - Successful request

`<span style="color: #ba372a;">201</span>` Created - Successful element creation request

`<span style="color: #ba372a;">204</span>` No Content - Successful request with an empty response

`<span style="color: #ba372a;">400</span>` Bad Request - Invalid request

`<span style="color: #ba372a;">404</span>` Resource is not found - The resource does not exist

`<span style="color: #ba372a;">422</span>` Unprocessable Entity - The request does not contain the

`<span style="color: #ba372a;">500</span>` Internal server error

</details>