Introduction

Free your Ebook collection!!!

Are you tired of being locked in to commercial Ebook software such as Apple iTunes (shudder), Google Books or Amazon Kindle? Would you like to take complete control of your Ebook collection and host it yourself, giving you the ability to serve whatever Ebooks you want at any time and at no extra charge? You’re not alone.

This is a short tutorial to show you how to host a Calibre library on a Linux server with the help of Google Drive and a nifty php program called COPS.
The general idea is that you will create a Calibre library on your local computer (or perhaps you already have an existing Calibre library) and then synchronise it using Google Drive (or you can use a cloud storage provider of your choice, the concept is the same).  In this way any changes you make to your Calibre library, now or in the future, will be automatically synchronised with cloud storage.  At your Linux server end, you will install a command line google drive client which will allow you to pull the synchronised Calibre library down to the server and keep up to date with any changes made on your local machine.  This synchronisation, using Google Drive, will help you to keep an up-to-date Calibre Library that has all your latest books, book covers and meta data on your Linux server.  The synchronised calibre library will be ‘served’ (made available on the web) using a program called COPS, which is able to serve a calibre library using a web based interface and also standard (ODPS) Ebook library interfaces.  This means you will be able to read your ebooks on any device with an internet browser or standard Ebook reader application.

 

You will be able to access your entire Ebook collection, anywhere and at any time.  Nobody will tell you which books you can host and which you cannot.  You won’t be locked in to any Ebook provider.  Your Ebook library will be free.

Careful about the ‘free’

Okay, so we will need to take a small precaution here.  Your Ebook collection is okay to be free to you, both in terms of liberty and cost, but you don’t want every man and his dog getting access to your Ebook collection.  This will be easily solved by creating a .htaccess file which you will place on your library’s web space.  This will allow you to create a simple username and password which will be required to access your library.  It just takes a few minutes and only needs to be set up once.  I will show you how.

A demonstration of what we’re trying to achieve

I often feel that it helps to be able to see and use an end product so that I can see where I’m going with something before I delve too far into it.  The user interface for our personal online Ebook collection will be provided by a little PHP program called COPS, and the author of this program provides a demonstration of this at his website, here.

A Couple of PreRequisites

In order to be able to synchronise files and host your own content you will of course need access to a few things first.  The pre-requisites for this tutorial are:

• Some web space, hosted on a Linux platform, which you either have direct access to (your server is at located at home, for example) or you have remote SSH (root) access to – for installation of your own programs and services.  Simple web space hosted by someone else is not good enough because those services won’t let you have control of the server itself.
• A Google Drive account.  You can just register a free one if you want; competition in cloud storage is rife these days, so even free accounts have enough storage to reliably synchronise thousands of Ebooks.

The Tutorial

Step 1 – Installing Google Drive on your local machine.

If you’re a Windows or Mac user this is easy.  Just visit https://tools.google.com/dlpage/drive and download the client for your computer.  Follow the installation instructions and choose a location on your local machine which you want to sync.  Create a folder in your local Google Drive called ‘Ebooks’.  You will put a Calibre library here later, and use Google Drive to synchronise it.  If you’re a Windows or Mac user, this step is now complete and you can skip forwards to step 2.  If you’re a Linux desktop user, read on…

If you’re a Linux user it’s not quite so easy.  Google have promised us an official Linux client for years, but so far it hasn’t been delivered and there is no indication as to when it will ever be delivered.  One google employee has even labelled the entire concept of automatic background synchronisation as fundamentally flawed, and I think she has a point.  She is the initial creator of a Google Drive command line client for Linux (called ‘drive’), which provides manual synchronisation capability.  This is the program you will install on the server later to pull down your Calibre library from Google Drive.  But you can also install it on your Linux desktop machines to push local changes to your Calibre library up to Google Drive.

To install the semi-official drive command line client for a Linux local machine, follow the same steps as in (3) and (4) for your server; the process is identical.

Step 2 – Installation & Setup of Calibre on your local machine

Calibre is a free, open-source, Ebook management tool.  Even if you do nothing else in this tutorial, this step alone will empower & transform your local Ebook collection.  You can manage the book covers, change the meta-data, sort your books using tagging or virtual libraries, etc.  If you just want to manage your Ebooks locally then Calibre is a fantastic tool.  It even has its own hosting tool to allow you to host a Calibre library online.  The tool we are going to use instead (COPS) is designed with light-weight servers in mind (suitable for a VPS), and that’s why I am not going to be covering the built in Calibre hosting here.

To install Calibre, go here and download the client that is appropriate for your operating system.  If you use Linux I still recommend that you fetch the program from the Calibre website as opposed to using apt-get.  In my experience the version in the standard repository for Ubuntu is way out of date.

I won’t cover set up of a Calibre library in any detail here.  That is a separate topic.  But you will find that it is pretty intuitive to set up a library, and there is extensive help documentation available.  During the set up of your library you will be asked to set a location for it:

Make certain that you set up your library inside the Ebooks folder you created on your local Google Drive.  This is how you will synchronise it to Google and to your server.  I called my Calibre library ‘BJH’.  Call yours whatever you like.

Once the library is set up you can add books by simply dragging them into the application.  There is a tool which can automatically fetch your book cover and meta data from the internet too – this is extremely useful.  I have one hint here; it is often better to ‘prepare’ your books for automatic meta-data collection by manually adding the ISBN into the meta-data editor yourself first.  This helps Calibre to find your book online in the first place, and it also helps it to get the cover for the correct edition.

Any changes you make to your Calibre library should be automatically synchronised with Google Drive.  Or, if you’re on Linux, you should manually run the “drive push” command described later.  You don’t have to waste all your day perfecting your Ebook collection right now if you don’t want to; just add some books in there to get started so that you can continue with the server setup.  You can perfect it as much as you like another time!

Step 3 – Installing GO on your server

If you want to run Google Drive on a Linux server (or on a local Linux machine), then you will be disappointed to learn that there is no official client for it.  There are a few options, both free and commercial, but I am only going to discuss Google’s semi-official command line client here, which is free.  I say “semi-official” because it was developed by a Google employee who has worked on a Linux client.  The command-line client is written in a programming language developed by Google called ‘GO’.  You will need to install a GO programming environment on your system so that you can fetch the source code for the drive client and build it on your system.  This is nowhere near as difficult as it sounds, and I’ll take you through all the steps:

Install GIT

You need GIT on your machine first, so that you can clone GIT repositories.  Open a command line (if installing on your server, you need to open an SSH connection to it) and type:

sudo apt-get install git

Follow the instructions.  Once done, you can check all is well by typing:

git --version

You should get back a version number for git.

Install Mecurial

sudo apt-get install git mercurial

Download GO & Extract

Visit the golang main page and access the downloads area. Select a stable version to download:

 

Upload the archive to your server, or you can use wget on your server to download it straight to your remote machine.  Open a terminal and ‘cd’ to the directory where you downloaded/uploaded the archive. e.g:

cd $HOME/Downloads

Then extract the go archive to /usr/local making certain to substitute “go1.4.linux-amd64” in the example below for the name of the archive you downloaded:

sudo tar -C /usr/local -xzf go1.4.linux-amd64.tar.gz

 Create a GO Workspace

Now you need to create a GO workspace.  This is a place where you will download 3rd party packages (the semi-official Google Drive program, in this example) and where you would create your own GO programs if you decided to experiment with GO yourself.  You can create the workspace anywhere you like.  In my example, I will create a workspace in $HOME/Programming/GO/ as follows:

mkdir -p $HOME/Programming/GO/src/github.com/bh4017

The reason for the [src] directory is that a GO workspace requires somewhere to put sourcecode.  The reason for the [github.com] directory underneath [src] is that you will often download the sourcecode for GO programs from github, and finally the reason for the [bh4017] directory is that I wanted a place to store my own GO programs.  bh4017 is my github.com id.  You should substitute ‘bh4017’ for your own github.com account name, or leave it out completely if you don’t intend to experiment with your own GO programs.

Inform the GO tool about your workspace

You need to tell GO where your workspace(s) are.  You can do it inside the terminal using an ‘export’ command, but the problem is that GO will forget about your workspace as soon as your terminal is closed.  To make GO ‘remember’ your workspace, you can remind it each time you log in by placing the ‘export’ commands inside your ‘.profile’ file.  This file is executed at each log-in and so GO will be automatically reminded of your workspace(s) every time you start Linux.  Open .profile in the editor of your choice.  In the following example I will use nano:

nano $HOME/.profile

Once you have opened your .profile, scroll to the bottom and add the following lines.  In the second and third ‘export’ lines, substitute my GO workspace path for the path you chose when you created your workspace.

# BEGIN GO
export PATH=$PATH:/usr/local/go/bin
export PATH=$HOME/Programming/GO/bin:$PATH
export GOPATH=$HOME/Programming/GO
# END GO

In order to make these changes go ‘live’ you need to log off and back on again so that .profile is executed.  You should do that now.

A good way to test your GO installation is to write the ubiquitous ‘hello world’ program.  There’s a tutorial available for this on the GO website.  If you don’t want to bother with that for now, then as a minimum you should open a terminal and type ‘go version’.  If successful, GO should return a version number to you.

Step 4 – Installing ‘drive’

Assuming you’re happy with your installation of GO, the following command should fetch the latest ‘drive’ source code and build it on your system:

go get -u github.com/odeke-em/drive/cmd/drive

No news is good news, so if your GO installation is working properly the terminal will return after a short while without any messages.  GO should have fetched the source code from github.com and built it for you.  To check that it did something, cd to $HOME/Programming/GO/bin (substituting for your own workspace path as created earlier).  Type ‘ls’ and if all is well you should see drive listed in there.

If your ‘export’ commands have been entered correctly you should be able to open a terminal at any time and simple type ‘drive’ to execute a google drive command.  Entering ‘drive’ on its own will bring up a usage response:

Step 5 – Synchronising your Ebook collection to webspace

First you need some webspace.  Instructions for this are outside the scope of this tutorial; presumably if you’re managing a linux server then you already know how to set up some web space.  For my own Ebook collection I used a subdomain called ‘library’, so my Ebook collection is now hosted at http://library.brianhoskins.uk.

Once you’ve got some web space, you need to find out where the space is located on your server.  For example, it could be located at /var/www/vhosts/yourdomain/sub-domain/ but the actual location will vary depending on your setup.  Once you know where your web space is located on your server, ‘cd’ to the location in a terminal and create a Google Drive folder using the ‘mkdir’ command.  You can name it whatever you want – I literally named mine “GoogleDrive”.

Once you have created your folder, you need to initialise it.  In your terminal, type the following:

drive init your-folder-name

If all goes well your terminal should respond with a url to obtain an authentication code.  This is to allow the drive client access to your Google Drive.  You need to use a browser to visit the provided link, and then you will be given an authentication code.  Paste this authentication code back into the terminal and press ‘enter’.

Once your folder has been initialised as a folder to manage using ‘drive’, ‘cd’ into it and type:

drive pull your-folder-name/Ebooks/your-library-name

drive should spend a few minutes resolving (comparing cloud to local) and then it should respond with a confirmation showing what it intends to download fresh from the cloud and, if this is not the first time you’ve pulled from Google (you are updating your Ebook collection) the files already downloaded which it intends to modify.  You are asked to accept, and then drive will pull down your Calibre library from the cloud to your server.  You now have a fully synchronised Calibre library on your Linux server!  In future, when you make changes to your library using Calibre on your local machine, you will simply allow Google Drive to synchronise them and then SSH into your server to run the ‘drive pull Ebooks’ command on your server again.  In this way, your server’s calibre library will be kept continually up to date with the books you add, delete, and maintain (covers, tags, meta data and all) on your local machine.

Note that if you are also using the linux drive client to synchronise a local calibre library up to the cloud, then you should use the ‘drive push’ command to synchronise local changes made to your Calibre library.

Step 6 – Installing & Configuring COPS on your server

Now that you have a way of maintaining a Calibre library on your local machine(s), synchronising them with Google Drive, and then pulling down synchronised changes to your server, you need a way of ‘serving’ your calibre library online.  This is what COPS does for us.  Fortunately, it’s extremely easy to install and set up!

Pre-Requisites

First of all, you need PHP 5.3/5.4 on your server and SQLite3.  My server is Ubuntu 12.04 and it wasn’t necessary for me to install or change anything on my system.  If yours is different, you should check which versions of these applications your server is running and update if necessary.

Downloading COPS

You can get COPS from github or you can get a zip archive from the author’s website.  I won’t link to the actual zip download because it may change depending on future updates to the software.  Visit the website I linked to, scroll to the very bottom, and under the heading “Download” you will find the latest zip file.

Installing COPS

Once you’ve downloaded the zip, installation is very straightforward.  Simply extract the contents of the zip to the root of your chosen webspace.  Once this is done, you should be able to browse your website and see a COPS interface.  It’ll report a bunch of errors because it can’t find your Calibre library – we’ll set that up in the next step.

Configuring COPS

First of all, enter the following commands:

mv config_local.php.example config_local.php
nano config_local.php

This should copy a sample setup into the COPS config file, and then open it for editing.

Find the line $config[‘calibre_directory’] =

Here you must enter the location of your Calibre directory on your server.  It is the directory underneath which the metadata.db file is to be found.  As an example, if you called your library ‘myBooks’, and placed it inside a folder structure called GoogleDrive/Ebooks/, then the config line would be:

$config['calibre_directory'] = './GoogleDrive/Ebooks/myBooks/';

Make CERTAIN that you include the final ‘/’, or COPS will not be able to find your Calibre metadata.db file and it won’t be able to serve your books!

Next, find the line $config[‘cops_title_default’]

This line simply sets the title bar.  Call it whatever you want – I changed mine to say library.brianhoskins.uk.

Once you’re happy, save the file and exit.

DONE!

That’s it, you should be done! Browse to your library webspace and COPS should be serving your Ebooks.  You can get to them via any modern browser.  You can also use an Ebook reader to view the books.  There are some dedicated Ebook readers for mobile platforms too – I use Moon+ reader pro on Android.  You can set up access to your library in these applications.

Setting Password Access

.htaccess

Okay, so now your library is working but every body on the planet can access it.  We’ll solve this problem in the next steps.

Open up a terminal and ‘cd’ to the root of your library website, where you installed COPS.  In there you will find a .htaccess file.  Open it up in a text editor.  Scroll to the bottom and find the following text:

#<FilesMatch "\.php$">
#AuthUserFile /path/to/file
#AuthGroupFile /dev/null
#AuthName "Acces securise"
#AuthType Basic
#Require valid-user
#</FilesMatch>

These are the entries which control password access.  At the moment they’re commented out, so password access is disabled.  Uncomment them by removing the ‘#‘ characters.

Now you need to point the .htaccess file to a password file called .htpasswd.  You haven’t created this file yet, but we’ll create it in a minute.  You need to place it somewhere where access is generally restricted (usually the root of webspace is).  Here’s a sample setup for you – tailor it you your specific requirements:

<FilesMatch "(index|feed)\.php">
AuthUserFile /var/www/vhosts/mydomain/sub-domain/.htpasswd
AuthGroupFile /dev/null
AuthName "SPEAK FRIEND AND ENTER"
AuthType Basic
Require valid-user
</FilesMatch>

.htpasswd

Now you need to actually create your .htpasswd file at the location where you told .htaccess it would be.  Create it and open it up in a text editor.

All you need to do now is add a user and password.  The password, though, should be encrypted.  The easiest way to go about this is to use an online tool, like this one.  Enter in your desired username and password.  The tool will then create the text you need to enter into your .htpasswd file.

FINISHED

Hopefully you’re done! If you now browse to your online library you should be prompted to enter a username and password.  Enter the plain text user and pass you set up earlier; the decryption is taken care of for you.

It’s a lot of fun managing your own online ebook repository.  Any questions, leave them in the comments and I’ll try my best to help!