Blockchain Indexers I: Introduction

Indexers have become a critical part of blockchain applications as without them, using applications built on the blockchain would be very slow. In this article, we will look at why it's important to build one and how you can go about it.

What is a Blockchain Indexer?

An indexer in layman's terms, is a program that makes it easier to query data on the blockchain. You see, blockchain is like a database that's open to everyone. It keeps growing with more transactions on the network.

However, this large database, due to its nature, is not very easy to query. You cannot simply SELECT * FROM ethereum.transactions WHERE sender='complexlity.eth' or some similar structured query method. You have to do the manual labor to get the data you want.

Indexers simply do this "manual labor" for you. They take the data from the blockchain as it happens and send it to a database that is easy to query. This is their sole purpose.

Types Of Indexers

When deciding to build an indexer for your application there are generally three types of indexers you have to consider when you need to get the data from the blockchain.

Proactive Indexers:

These are where you have to get all the data from the blockchain before your application even launches. "Proactive" because it means you have to do most of the work way before launch and continue pulling information over time. Some examples of this kind of application are:

1. You're building an app that lists all Crypto Punk NFTs, their price, owner, and bids and enables users to buy them.

If you're working on such an app, you have to go on the Crypto Punk Mint contract and pull the data for the minted NFTs, current owners, transfer or ownership history, etc. All of this has to be done for your app to even work initially.

2. You're building an application that shows all the users who have launched a token with Clanker

To make this kind of application, you'd have to go through the clanker contracts (Currently has V1, V2, and V3) and pull all the launch token data from the blockchain.

Active Indexers:

These are the kind of indexers where you write the contracts to your Dapp and make these indexers pull information for Day 1. The difference between Active and Proactive has to do with when you start pulling data. This is the most common type since most Dapps have custom contracts and the responsibility to show this information in a meaningful way to the user often relies on the Dapp builder and not any third party. Some examples of this kind of application are:

1. You're building Bountychain, an app that lets users create and complete bounties with a decentralized escrow system.

The contracts needed to make the app work need to be indexed from Day 1. Bounties created directly outside the Dapp still become visible in the Dapp.

2. You're building Stoke Fire, A game built on base where users grow their villages and compete with each other to see who can grow the most while hindering/destroying the other village for points.

Again, for this application to work properly, you need every activity carried out on the blockchain to reflect actively in the Dapp.

Reactive Indexers:

These are the types of indexers that are only linked and run on certain actions carried out on your Dapp. This kind of activity includes:

• Only running for a particular user when they sign up.
• Only running when a user goes to a certain page, etc. For these types of indexers, it is simply not viable to pull the entire data from the blockchain since you would likely not need to store all that information (the database has costs) so it's "Get what I need when I need it". Some examples of this kind of application are:

1. You're building an app that shows users' wallet balances over time. For such an app, you simply cannot pull the balances from every account on the blockchain since you do not need such data.

What you could do is run an indexer for a particular user the first time they sign up and store the balance history in the database. This way you only store information for users that you need to and not the entire blockchain. If you don't have a "signup" system, you could also run this indexer the first time someone navigates to the page for a particular wallet. Beyond that, indexing from the last point becomes faster and easier on consequent navigations.

2. You're building Interface, a social app that lets you see token purchases and mint transactions from your friends. Again, it is simply not feasible to pull the entire blockchain information. It has a signup system so you can rely on two modes:

• Pull all information for a user when they sign up
• Pull all transaction information for a user when other users navigate to their page. This is also performance efficient because if the user eventually signs up, you don't have to pull the transactions from scratch but from the last point.

NOTE: For all types of indexers, you do not stop indexing after it is done. The indexers have to keep being active as long as the app exists else the blockchain leaves you behind. So you either have to run as background jobs at regular intervals, synchronize the data on request, or have them actively pull information as they happen. Either way, they must remain active or your data becomes stale.

Conclusion

Blockchain indexers in summary are programs/applications that get data from the chain and make it easier to query and easier to read.

In the next article, we will go practical and build a fully working indexer.

Info

Dune has built on SQL syntax and using their platform, you can actually SELECT * FROM ethereum.transactions..... They have indexed entire chains similar to Moralis(which we will discuss in the next article)