Bot

​

Overview

• Bot traffic identification
• Search engine crawler handling
• Automated traffic filtering

​

Installation

import "github.com/go-mizu/mizu/middlewares/bot"

​

Quick Start

app := mizu.New()

app.Use(bot.New())

app.Get("/", func(c *mizu.Ctx) error {
    if bot.IsBot(c) {
        return c.Text(200, "Hello, bot!")
    }
    return c.HTML(200, fullPage)
})

​

Configuration

​

Options

​

Examples

​

Detect Bots

app.Use(bot.New())

app.Get("/", func(c *mizu.Ctx) error {
    if bot.IsBot(c) {
        // Serve simplified content
        return c.Text(200, "Content")
    }
    return c.HTML(200, richContent)
})

​

Block Bad Bots

app.Use(bot.New(bot.Options{
    Block:     true,
    AllowGood: true, // Allow Google, Bing, etc.
}))

​

Custom Patterns

app.Use(bot.New(bot.Options{
    Patterns: []string{
        "MyCustomBot",
        "InternalCrawler",
    },
}))

​

Bot-Specific Routes

app.Get("/sitemap.xml", func(c *mizu.Ctx) error {
    return c.XML(200, sitemap)
})

app.Get("/", func(c *mizu.Ctx) error {
    info := bot.Info(c)
    if info.IsSearchEngine {
        // SEO-optimized content
    }
    return c.HTML(200, content)
})

​

API Reference

​

Functions

// New creates bot detection middleware
func New(opts ...Options) mizu.Middleware

// IsBot returns true if request is from bot
func IsBot(c *mizu.Ctx) bool

// Info returns detailed bot information
func Info(c *mizu.Ctx) *BotInfo

​

BotInfo

type BotInfo struct {
    IsBot          bool
    IsGoodBot      bool
    IsSearchEngine bool
    Name           string
    Category       string
}

​

Known Good Bots

• Googlebot
• Bingbot
• Slurp (Yahoo)
• DuckDuckBot
• Baiduspider

​

Technical Details

​

User-Agent Pattern Matching

• Default Patterns: Includes search engines (Googlebot, Bingbot), social bots (Facebook, Twitter), SEO tools (Semrush, Ahrefs), and HTTP clients (curl, wget)
• Pattern Compilation: All patterns are compiled into a single regex at initialization for O(n) matching
• Case Insensitive: Matching is case-insensitive to handle various User-Agent formats

​

Bot Categories

• search: Googlebot, Bingbot, Yandexbot, Baiduspider, DuckDuckBot
• social: Facebook, Twitter, LinkedIn, Pinterest
• seo: Semrush, Ahrefs, Majestic, MJ12bot
• tool: curl, wget, Python requests, Go HTTP client
• crawler: Generic crawlers, spiders, scrapers

​

Context Storage

​

Allowlist/Blocklist Logic

1. If AllowedBots is set, only listed bots are allowed (allowlist mode)
2. If BlockedBots is set, only listed bots are blocked (blocklist mode)
3. If neither is set, all detected bots are blocked

​

Best Practices

• Allow search engine bots for SEO
• Serve simplified content to bots
• Log bot traffic for analysis
• Rate limit suspicious bots

​

Testing

​

Related Middlewares

• ratelimit - Rate limiting
• ipfilter - IP filtering
• xrequestedwith - AJAX detection