APIs: why they are useful, and why browser-based plugins that sidestep them aren’t always a good idea

While writing a thread on Twitter about Ottomated’s NoFT plugin (a browser plugin for auto-blocking NFT users), I thought it would be a good idea to write an article explaining about APIs. The NoFT plugin doesn’t use the Twitter API, which comes with problems and risks for users, but the reasons for that need a lot of explaining.

Table of Contents

· What are APIs?
∘ Library Analogy
· Why do we have APIs?
∘ Efficiency
∘ Standardisation
∘ Tracking
∘ Decoupling the frontend from the backend
· Why aren’t browser-based plugins that sidestep APIs always a good idea?
∘ When it can be a good idea
∘ When it’s a bad idea
∘ What’s the risk of not using an API if an application or website has one?
∘ The risk for developers
∘ The risk for users
· Is it possible to make a browser plugin that also uses a website’s API?
∘ API Keys

What are APIs?

Application Programming Interfaces, or APIs, are effectively a way of communicating with a piece of software or a website. Throughout this article, I will use the librarian analogy below to illustrate the concept.

Library Analogy

An API is like a librarian, who provides an interface between you and a library full of books: you ask the librarian for a particular book, and the librarian finds it and gives it to you, then notes down that you have taken the book. You ask the librarian to put a book back and they find the right place for it to be and put it there, then note down that the book has been returned.

Why do we have APIs?

Efficiency

APIs are very efficient. The librarian knows where every book is stored, and knows the layout of the library. Asking the librarian to find a book is going to be far quicker than trying to find the book yourself.

This is the same in code; if a website or application has an API, the methods of that API (getting some data, or posting some data to it) can be optimised for efficiency and reliability, whereas if every developer is making their own way of asking for data, each one will vary wildly in performance.

On top of that, when it comes to e.g. scraping data from webpages, apps that do this often end up requesting far more data than they need to. For instance: if a website shows the time, and you request the time via an API, it’ll probably give you a short string denoting the time in response.

If you scrape the webpage, you must download all of the website’s HTML, CSS pages, and so on — a whole ton of extra, unnecessary information you didn’t need, which also places strain on the website’s bandwidth.

Standardisation

Hundreds of people might use the library every hour. Instead of having everyone running around using their own unique method of book-finding, having them ask a librarian — who uses a standard method for finding any given book — results in much more reliable finding of books, much less chaos, and far faster retrieval of books.

APIs are the same. Not only does having a standardised set of ways to retrieve or post data to an application make it more efficient, but it also makes life easier for app developers, who know exactly what format the returned data will be in, exactly what format their request should be in, and so on.

Tracking

The library needs to know which books are on loan, how many books it has available, who has borrowed which books, etc.

If everybody is taking books off the shelves themselves, it’s very difficult for the library to find out which books are on loan or which are returned. If everyone is asking the librarian to find books instead, the librarian can track this information as they perform the actions, allowing the library to keep its status up to date easily and efficiently. This helps the library to ensure people aren’t taking out too many books at once, to find out who took a book if it isn’t returned, and so on.

With APIs, it’s exactly the same. By having an API, a website or application can track who is calling which endpoints, how many requests are being made by a specific user, enforce limits on how many requests can be made, and more. This helps websites to manage their workload, prevent abuse of their services, and gather information about which of their services are most used.

Decoupling the frontend from the backend

The librarian uses indexes and databases to know which book is stored where. If users are looking themselves, they depend on remembering which bookcase is where and other irrelevant information: if the layout of the library changes, the librarian still knows exactly where everything is, but the user will be confused and wander around searching for a book that was previously easy to find.

This principle is crucial, not only for APIs but for software in general: you never want backend code relying on frontend code, as there is no good reason for it to do that, and it causes bugs whenever the frontend layout changes.

Browser plugins that parse webpages often fall victim to this problem: their ability to function depends on the exact layout of the webpage, and when that layout changes, the app can often break or end up with bugs. If you use an API to contact that webpage, the API doesn’t change when the frontend changes, and so your app will continue working as normal.

Why aren’t browser-based plugins that sidestep APIs always a good idea?

There are a few reasons for this, but firstly, let’s look at the cases where it *is* a good idea (or at least, is not a bad idea).

When it can be a good idea

Suppose you want to make an application that, every time it sees a post from a Facebook user wearing a Trump hat in their profile picture, it replaces their profile picture in your browser with that of a clown (if there’s really any difference, anyway).

Your application — when the user is browsing Facebook — would parse the webpage, find the elements it wants to replace, and then replace them in the browser — i.e. on the client side. Facebook would never know the app was even doing anything; everything is being done in the user’s browser. As such, a browser app like this presents no problems, because:

• It is not performing any server-side actions (e.g. posting something to a Facebook page or automating actions that Facebook’s servers need to perform).
• It isn’t requesting any data the user didn’t already request, because it’s only reading the webpage when the user already went to the webpage, i.e. the webpage source was already downloaded. As such it’s placing no extra strain on Facebook’s servers.

When it’s a bad idea

There are two situations when it’s a bad idea, very much the opposite of the two bullet points above:

1. When you are automating something the server has to do (e.g. an app that scrapes the page and then blocks people for you via the webpage’s own blocking system).
2. When you are scraping a webpage without the user looking at it (e.g. a bot that goes to websites to scrape their webpage contents), or automating loading extra elements on a page that the user wouldn’t otherwise be looking at.

You can more or less think of this as: if it’s something you should be doing via an API, you should not be doing it by scraping the page. Many browser extensions *do* scrape the page; sometimes because the developer didn’t want to learn how to use the API, sometimes because they want to get around the API’s rate limits, or other reasons.

The problem with scraping the page, and not using the API, partially falls on the developer but also falls on users, as explained below.

What’s the risk of not using an API if an application or website has one?

Most websites or applications which use APIs have them in place for a good reason: to track request numbers, for efficiency, and to prevent abuse. As such, said websites and applications do not like anything that tries to sidestep this.

The risk for developers

The risk for developers is that such an app could get banned from other stores if the website whose API it sidesteps complains, though the likelihood of this actually happening is somewhat questionable.

The bigger, and more relevant, risk is that the app will be prone to breaking. An application which sidesteps the API is relying on parsing the webpage to perform its actions, for example: let’s say your app replaces Facebook profile pictures with Trump hats with clown pictures.

That application is going to be parsing the Facebook page, looking for the specific sections of the page in which profile pictures are shown. What do you do if Facebook updates its page layout and that changes? Your whole app has now broken until you hard-code in the new layout parsing. Using an API avoids this problem, because you have decoupled the backend code from the frontend appearance of the website.

The risk for users

When an app doesn’t use a website or application’s API, the user will tend to be liable for whatever that app is doing.

For instance, let’s say you add an app to your browser that automatically blocks people in your Twitter feed who say they like pineapple on pizza. Since the app is scraping the page, and not using the API, Twitter has no clue that an app is doing this blocking for you; all it knows is that your browser is sending requests to block people by pressing the block buttons on the page.

However, Twitter does know that such activity isn’t usually human. If you scroll down a page of #pineappleonpizza search results and block 10 people in 0.2 seconds, Twitter’s systems know full well that a human is not capable of blocking that many people in that timeframe, and thus immediately recognises that you are using an automated app that is scraping the webpage. It can also tell this from the different headers given in the webpage request, or different patterns of interactivity on the webpage vs normal activity; such systems are usually quite complex.

Is it possible to make a browser plugin that also uses a website’s API?

Yes and no.

You can do it, but you’d also need a server to make requests on behalf of your app’s users — meaning your browser plugin is a convenience only (as if you have a server, you can make a website which will achieve the same thing for users).

If you’re not using a server, it won’t be possible, because using any API comes with security considerations which a purely client-side solution cannot solve.

API Keys

When you register an app to use a website’s or other app’s API, you will generally have two types of credentials (there is a third but there is no need to cover it here):

• API keys, keys that identify your app when making API calls.
• Access tokens, allowing your app to make requests on behalf of specific users.

It should be obvious that user access tokens must be kept secret from everyone except the app and the user. API keys, however, must be kept secret from everyone except the app itself. If your API keys were discovered by anyone else, they could use them to make dodgy requests on behalf of your app which you didn’t intend it to make.

As a result, you can’t implement a purely browser-based plugin whilst using an API. How is the browser plugin going to make API requests? It has to know the API keys in order to do that, and if the API keys are known to your user’s browser, they are easy for any knowledgeable user to discover, immediately compromising your app’s security.

The only way to avoid this is to use a server, which acts as an intermediary; your user’s browser contacts your server, and on your server, your app performs the API request on behalf of the user and sends back the response. This way, only your server ever sees the API keys, and thus they remain secret.