Chaos - 💧 Mizu

Overview

The chaos middleware injects failures and latency into your application for chaos engineering and resilience testing. Use it to verify how your system handles errors and delays.

Installation

import "github.com/go-mizu/mizu/middlewares/chaos"

Quick Start

app := mizu.New()

// Inject 10% errors with random latency
app.Use(chaos.WithOptions(chaos.Options{
    Enabled:    true,
    ErrorRate:  10,
    LatencyMin: 100 * time.Millisecond,
    LatencyMax: 500 * time.Millisecond,
}))

Configuration

Option	Type	Default	Description
`Enabled`	`bool`	`false`	Enable chaos injection
`ErrorRate`	`int`	`0`	Percentage of requests to fail (0-100)
`ErrorCode`	`int`	`500`	HTTP status code for errors
`LatencyMin`	`time.Duration`	`0`	Minimum latency to inject
`LatencyMax`	`time.Duration`	`0`	Maximum latency to inject
`Selector`	`func(*mizu.Ctx) bool`	-	Filter which requests to affect

Examples

Error Injection

// Fail 20% of requests with 500 error
app.Use(chaos.Error(20, 500))

Latency Injection

// Add 100-500ms latency to all requests
app.Use(chaos.Latency(100*time.Millisecond, 500*time.Millisecond))

Combined Chaos

app.Use(chaos.WithOptions(chaos.Options{
    Enabled:    true,
    ErrorRate:  5,          // 5% errors
    ErrorCode:  503,        // Service Unavailable
    LatencyMin: 50 * time.Millisecond,
    LatencyMax: 200 * time.Millisecond,
}))

Selective Chaos (Path-based)

app.Use(chaos.WithOptions(chaos.Options{
    Enabled:   true,
    ErrorRate: 50,
    Selector:  chaos.PathSelector("/api/orders", "/api/payments"),
}))
// Only affects /api/orders and /api/payments

Selective Chaos (Method-based)

app.Use(chaos.WithOptions(chaos.Options{
    Enabled:   true,
    ErrorRate: 10,
    Selector:  chaos.MethodSelector("POST", "PUT", "DELETE"),
}))
// Only affects write operations

Header-triggered Chaos

app.Use(chaos.WithOptions(chaos.Options{
    Enabled:   true,
    ErrorRate: 100,
    Selector:  chaos.HeaderSelector("X-Chaos-Test"),
}))
// Only affects requests with X-Chaos-Test header

Dynamic Control

controller := chaos.NewController()

app.Use(controller.Middleware())

// Control chaos via admin endpoints
app.Post("/admin/chaos/enable", func(c *mizu.Ctx) error {
    controller.Enable()
    return c.Text(200, "Chaos enabled")
})

app.Post("/admin/chaos/disable", func(c *mizu.Ctx) error {
    controller.Disable()
    return c.Text(200, "Chaos disabled")
})

app.Post("/admin/chaos/config", func(c *mizu.Ctx) error {
    rate, _ := strconv.Atoi(c.FormValue("error_rate"))
    controller.SetErrorRate(rate)

    latencyMs, _ := strconv.Atoi(c.FormValue("latency_ms"))
    controller.SetLatency(
        time.Duration(latencyMs/2)*time.Millisecond,
        time.Duration(latencyMs)*time.Millisecond,
    )

    return c.Text(200, "Configured")
})

app.Get("/admin/chaos/status", func(c *mizu.Ctx) error {
    return c.JSON(200, map[string]bool{
        "enabled": controller.IsEnabled(),
    })
})

Environment-Based

// Only enable in testing environment
if os.Getenv("CHAOS_ENABLED") == "true" {
    rate, _ := strconv.Atoi(os.Getenv("CHAOS_ERROR_RATE"))
    app.Use(chaos.WithOptions(chaos.Options{
        Enabled:   true,
        ErrorRate: rate,
    }))
}

Custom Selector

app.Use(chaos.WithOptions(chaos.Options{
    Enabled:   true,
    ErrorRate: 30,
    Selector: func(c *mizu.Ctx) bool {
        // Only affect non-admin users
        userRole := c.Get("user_role")
        return userRole != "admin"
    },
}))

Per-Route Chaos

// Global middleware (disabled)
app.Use(chaos.New())

// Enable chaos for specific route
app.Get("/test/chaos",
    chaos.Error(50, 500),
    func(c *mizu.Ctx) error {
        return c.Text(200, "Success!")
    },
)

Simulate Downstream Failures

app.Get("/api/external", func(c *mizu.Ctx) error {
    // Middleware adds latency/errors before reaching here
    result, err := callExternalService()
    if err != nil {
        return c.JSON(500, map[string]string{"error": err.Error()})
    }
    return c.JSON(200, result)
})

Selectors

Selector	Description
`PathSelector(paths...)`	Match specific URL paths
`MethodSelector(methods...)`	Match HTTP methods
`HeaderSelector(header)`	Match requests with header

Controller Methods

func NewController() *Controller
func (c *Controller) Enable()
func (c *Controller) Disable()
func (c *Controller) IsEnabled() bool
func (c *Controller) SetErrorRate(rate int)
func (c *Controller) SetErrorCode(code int)
func (c *Controller) SetLatency(min, max time.Duration)
func (c *Controller) SetSelector(selector func(*mizu.Ctx) bool)
func (c *Controller) Middleware() mizu.Middleware

API Reference

func New() mizu.Middleware
func WithOptions(opts Options) mizu.Middleware
func Error(rate int, code int) mizu.Middleware
func Latency(min, max time.Duration) mizu.Middleware

// Selectors
func PathSelector(paths ...string) func(*mizu.Ctx) bool
func MethodSelector(methods ...string) func(*mizu.Ctx) bool
func HeaderSelector(header string) func(*mizu.Ctx) bool

Safety

Always protect chaos endpoints:

// Restrict chaos control to admin
admin := app.Group("/admin")
admin.Use(basicauth.New(basicauth.Options{
    Users: map[string]string{"admin": "secret"},
}))

// Add chaos control routes to admin group
admin.Post("/chaos/enable", enableChaosHandler)

Technical Details

Architecture

The chaos middleware implements failure injection through a middleware chain pattern, supporting both static and dynamic configuration:

Static Configuration: WithOptions(), Error(), and Latency() create middleware with fixed settings
Dynamic Configuration: Controller provides thread-safe runtime configuration changes

Implementation Details

Random Number Generation:

Uses math/rand instead of crypto/rand for performance (intentionally weak RNG)
Error injection: Generates random number 0-99 and compares against error rate percentage
Latency injection: Calculates random duration between min and max using rand.Int63n()

Latency Calculation:

latency = LatencyMin + rand.Int63n(LatencyMax - LatencyMin)

Request Flow:

Check if chaos is enabled
Apply selector filter (if configured)
Inject latency (if configured)
Inject error based on probability (if configured)
Pass to next handler (if no error injected)

Selector Functions:

PathSelector: Creates a map of paths for O(1) lookup
MethodSelector: Creates a map of HTTP methods for O(1) lookup
HeaderSelector: Checks header presence using standard library

Controller Thread Safety:

Options are read atomically during request processing
Enable/Disable uses boolean flag
Configuration methods update fields directly (suitable for controlled admin access)

Performance Considerations

Selector maps provide constant-time path/method matching
No mutex locking on hot path for static configuration
time.Sleep() blocks the handler goroutine (intentional for testing)
Random number generation is non-cryptographic for speed

Best Practices

Never enable in production without safeguards
Use selectors to limit scope
Start with low error rates
Monitor during chaos testing
Use header-triggered chaos for CI/CD tests
Implement circuit breakers in your clients

Testing

The chaos middleware includes comprehensive test coverage for all features:

Test Case	Description	Expected Behavior
`TestNew`	Default middleware creation	Returns 200 OK (chaos disabled by default)
`TestError`	Error injection with 100% rate	Returns 503 Service Unavailable
`TestLatency`	Latency injection (50-100ms)	Request takes >= 50ms to complete
`TestWithOptions_Disabled`	Disabled chaos with 100% error rate	Returns 200 OK (no error injection)
`TestWithOptions_Selector`	Custom selector for specific path	Injects error on `/chaos`, allows `/normal`
`TestController`	Controller enable/disable	Correctly toggles enabled state
`TestController_Middleware`	Controller middleware integration	Injects errors only when enabled
`TestController_SetErrorCode`	Custom error code via controller	Returns 502 Bad Gateway
`TestPathSelector`	Path-based selector	Matches `/api/chaos`, ignores `/api/normal`
`TestHeaderSelector`	Header presence selector	Selects requests with `X-Chaos` header
`TestMethodSelector`	HTTP method selector	Matches POST, ignores GET

timeout - Request timeout
circuitbreaker - Circuit breaker
recover - Panic recovery

​Overview

​Installation

​Quick Start

​Configuration

​Examples

​Error Injection

​Latency Injection

​Combined Chaos

​Selective Chaos (Path-based)

​Selective Chaos (Method-based)

​Header-triggered Chaos

​Dynamic Control

​Environment-Based

​Custom Selector

​Per-Route Chaos

​Simulate Downstream Failures

​Selectors

​Controller Methods

​API Reference

​Safety

​Technical Details

​Architecture

​Implementation Details

​Performance Considerations

​Best Practices

​Testing

​Related Middlewares