Index up to 4x Faster with SubQuery's Parallelised Processing

SubQuery is now a lot faster with support for worker threads. This feature can speed up your indexing by up to 4 times!

At SubQuery, we're always focusing our efforts towards improving indexing speed. Our aim is to be the fastest multi-chain decentralised indexer! Today we take a big step forward with support for parallelisation using worker threads. This feature will move block fetching and block processing into its own worker thread, which results in increased efficiency of indexing. It should speed up indexing by up to 4 times (depending on the particular project).

Why worker threads?

NodeJs is by default a single-threaded programming language using the event loop execution model, which means that NodeJs applications can run one function at a time.

In case of intensive CPU processing, the event loop gets blocked and we can not execute other instructions until the heavy processing power function is completed. Therefore, we started using NodeJs worker threads to run some instructions in multiple threads.

How Do I Start?

The feature is disabled by default while we are still testing it. However, we are so excited we wanted to let you know about it and get your feedback. You can easily enable it using the -workers=<number> flag. Note that the number of available CPU cores strictly limits the usage of worker threads. For now, it is only available for Substrate and Cosmos and will soon be integrated for Avalanche.

Our hosted service doesn't offer this feature currently and we expect to add support for paid and partner projects shortly.

Find out more here and if you have any feedback, please let us know!

https://github.com/subquery/subql/pull/1103

About SubQuery

SubQuery is a blockchain developer toolkit facilitating the construction of Web3 applications of the future. A SubQuery project is a complete API to organise and query data from Layer-1 chains. Currently servicing Polkadot, Substrate, Avalanche, Terra, and Cosmos (starting with Juno) projects, this data-as-a-service allows developers to focus on their core use case and front-end without wasting time building a custom backend for data processing activities. In the future, the SubQuery Network intends to replicate this scalable and reliable solution in a completely decentralised manner.

​​Linktree | Website | Discord | Telegram | Twitter | Matrix | LinkedIn | YouTube

You might also like

Blog Feature Image

The Ethics and Benefits of Decentralised AI

Who controls the AI that shapes our world today? And should it be different? Artificial Intelligence (AI) has woven itself into the fabric of our lives, from personalised recommendations, to helping us interact with our favourite web3 project, even to advanced medical diagnostics. Yet, as AI becomes more central to how we live and work, the questions of control, fairness, and accessibility grow more pressing.  Centralised AI systems, controlled by the largest corporations ever formed, dominate
Tue Nov 19 2024
Read more
Blog Feature Image

Behind the Node: Meet Node.Monster

In our Behind the Node series, we dive into the world of SubQuery Network's Node Operators - the vital players that help power our globally distributed and decentralised ecosystem. This series allows the community to learn more about who they are delegating their SQT tokens to, and what makes these operators stand out. In this edition, we introduce Node.Monster, a team that's been involved in blockchain from its early days and is now a respected Node Operator committed to providing secure, high
Mon Nov 11 2024
Read more
Blog Feature Image

Real-World Examples of AI Applications in Web3

As Web3 technology evolves, industries are adopting self-contained AI agentic applications—referred to as AI Apps—to improve user experiences, enhance operational efficiency, and enable more seamless interaction with decentralised platforms.  These AI Apps, or autonomous agents, are programmed to perform specific functions with minimal human intervention, making them ideal for Web3 environments where users demand decentralisation, privacy, and efficiency.  Here’s a look at how AI Apps are tran
Wed Nov 13 2024
Read more