My Biggest Project So Far: Building a Private Manga Scraper Telegram Bot

BY: Arihant DATE: January 02, 2026

(Why this project changed how I understand systems, the internet, and engineering)

1. My Mindset Before Starting

I’m not someone who is comfortable using tools as black boxes.

If I don’t understand how something works internally, I feel limited using it.
Because of this, I naturally end up rebuilding things — not to reinvent the wheel, but to understand why the wheel works.

This mindset is what led to my biggest project so far:
a fully private, closed-source Manga Scraper Telegram Bot.

The goal was never “just make a bot”.
The goal was to understand systems end-to-end — from how websites behave, to how networks respond, to how backend services scale, to how long-running programs survive in production.

2. High-Level Overview of the System

At a user level, the bot works like this:

  1. User enters the name of a manga

  2. Bot asks which source/site to use

  3. Results are scraped and shown using pagination

  4. User selects the manga

  5. Chapters are fetched and listed

  6. Selected chapters are processed and delivered as:

    • PDF (with multiple customization options)

    • CBZ format

That’s the simple explanation.

The real complexity is everything underneath.

3. Architecture & Design Philosophy

From day one, I focused on:

  • Object-oriented design

  • Clean separation of concerns

  • Reusability

  • Maintainability

Every major part of the system was isolated:

  • Scrapers

  • Database layer

  • Queue system

  • Telegram UI logic

  • State management

  • Logging

  • Media processing

I built a consistent and concise file structure so that:

  • Any component could be reused independently

  • Adding new sites wouldn’t break existing logic

  • Debugging wouldn’t become painful as the codebase grew

This wasn’t accidental — early mistakes taught me how fast things break when structure is ignored.

4. Async-First Approach (The Core of the System)

One of the biggest challenges early on was blocking behavior.

Scraping, downloading, image processing, PDF generation — everything can block if handled incorrectly.

So I rebuilt the system with:

  • Heavy use of asyncio

  • Fully parallel execution wherever possible

  • No long-running blocking tasks in the main flow

Over time, I learned:

  • How concurrency actually works

  • Where race conditions happen

  • Why naïve async usage still blocks systems

  • How to manage shared state safely

This single decision shaped the entire system’s reliability.

5. Scraping Multiple Websites (Reality of the Web)

This project exposed me to the real diversity of the internet.

Not all websites are the same:

  • Some had no APIs

  • Some used REST

  • Some used GraphQL

  • Some were protected by Cloudflare

  • Some relied heavily on JavaScript rendering

To handle this:

  • I implemented traditional HTML scraping where possible

  • Used REST & GraphQL when available

  • Used Selenium selectively for JS-blocked sites

  • Learned how Cloudflare detects traffic

  • Learned what actually triggers blocks

  • Learned how authentication, headers, cookies, and sessions work

This gave me a deep understanding of internet architecture, not just scraping tricks.

6. Telegram UI, API & Internals

Telegram was not treated as a black box.

I learned:

  • How Telegram’s UI logic works

  • Pagination handling

  • Button callbacks and state persistence

  • How to store short, secure state data using hashes in buttons

At one point, instead of relying fully on existing wrappers,
I built my own Telegram client to interact with the raw Telegram API, just to understand:

  • How messages are actually sent

  • How updates flow

  • How wrappers abstract complexity

This significantly improved how I design client-server interactions.

7. State Management & Concurrency

With multiple users interacting simultaneously, I needed:

  • Live state tracking

  • Short-term concurrent state storage

  • Minimal memory overhead

So I designed:

  • A custom state system to store concise, relevant live data

  • Clear lifecycle management of user interactions

  • Isolation between concurrent users

This helped avoid data leaks, race conditions, and UI mismatches.

8. Databases & Data Handling

Different parts of the system needed different storage models.

So I worked with:

  • JSON for lightweight, structured data

  • MongoDB for flexible document-based storage

  • PostgreSQL for relational and structured data

Instead of directly using drivers everywhere, I created:

  • A custom database abstraction layer

  • Consistent interfaces for reads/writes

  • Easy future migrations or replacements

This taught me why abstraction matters in backend systems.

9. Media Processing & Advanced Features

Once manga data was fetched, the real processing began.

I implemented:

  • Image compression to keep PDFs manageable

  • Password-protected PDFs

  • Custom banners inside PDFs

  • Clickable links embedded inside images

  • Uniform PDF width handling

  • CBZ formatting

These weren’t cosmetic features — they required understanding:

  • Image formats

  • PDF internals

  • Processing pipelines

  • Memory management

I also added premium-style features, subscriptions, and access-based logic.

10. Queues, Background Tasks & Reliability

Long tasks cannot block user interaction.

So I built:

  • A custom queue system

  • Background task execution

  • Subscription-based job handling

This ensured:

  • Smooth UX

  • Fault tolerance

  • Controlled resource usage

11. Logging, Debugging & Observability

As the system grew, debugging became harder.

Instead of relying only on default logging, I:

  • Created custom logging methods

  • Designed logs for actual debugging, not noise

  • Tracked execution paths and failures clearly

Every issue I faced, I didn’t just patch —
I built a systematic solution.

12. Deployment, Docker & Linux

To run this in production, I learned:

  • Docker & containerization

  • How containers actually isolate environments

  • How services run on VPS servers

  • Linux filesystem, processes, permissions

  • Running background scripts reliably

This gave me a real understanding of how code lives beyond localhost.

13. Version Control & Engineering Discipline

Throughout the project:

  • I used Git seriously

  • Maintained clean commits

  • Structured code for long-term maintenance

This project taught me engineering discipline, not just coding.

14. What This Project Gave Me

This project gave me:

  • End-to-end system thinking

  • Confidence in building complex systems

  • Deep understanding of:

    • Internet architecture

    • Backend systems

    • Async programming

    • Databases

    • Deployment

    • APIs

    • UI logic

  • A clear vision of what I can build next

Most importantly, it changed how I approach problems.

15. Final Note

This project is private and closed-source, not because it’s secret —
but because the real value lies in what it taught me, not just the code itself.

This document exists because a single LinkedIn post isn’t enough to explain a journey like this.

If you’ve read this far, you understand my mindset:
I don’t just use systems — I understand them.

0 COMMENTS READ MORE »