日志洞察与分析
AWS 监控与故障排查
John Q. Martin
Principal Consultant
Logs Insights 查询结构
查询语言：同一查询的三种写法
CloudWatch Logs Insights QL
fields @timestamp, @message, level
| filter level = "ERROR"
| filter @timestamp > ago(1h)
| sort @timestamp desc
| limit 20
OpenSearch PPL
source = `/aws/lambda/my-function`
| where level = 'ERROR' and @timestamp > ago(1h)
| fields @timestamp, @message, level
| sort - @timestamp
| head 20
查询语言：OpenSearch SQL
OpenSearch SQL
SELECT `@timestamp`, `@message`, level
FROM `/aws/lambda/my-function`
WHERE level = 'ERROR'
AND `@timestamp` > ago(1h)
ORDER BY `@timestamp` DESC
LIMIT 20
- 标准 SELECT / FROM / WHERE 语法
- 可在日志组间执行 JOIN
- 选用最适合您团队的方式
场景 1：定位错误激增
fields @timestamp, @message, level
| filter level = "ERROR"
| stats count() as error_count by bin(5m)
| sort @timestamp desc
增强：按错误类型分组
fields @timestamp, error_type, error_message
| filter level = "ERROR"
| stats count() as count by error_type, bin(5m)
| sort count desc
场景 2：API 端点变慢
fields @timestamp, endpoint, response_time, status_code
| filter response_time > 1000
| stats avg(response_time) as avg_time,
max(response_time) as max_time,
count() as slow_requests
by endpoint
| sort avg_time desc
深入排查
fields @timestamp, endpoint, response_time, user_id, request_id
| filter endpoint = "/api/users" and response_time > 1000
| sort response_time desc
| limit 20
场景 3：身份验证失败
fields @timestamp, user_id, ip_address, action
| filter action = "login_failed"
| stats count() as failed_attempts by user_id, ip_address
| sort failed_attempts desc
| limit 50
基于时间的分析
fields @timestamp, user_id, ip_address
| filter action = "login_failed"
| stats count() as attempts by ip_address, bin(1h)
| filter attempts > 10
| sort attempts desc
场景 4 与 5：数据库超时与内存泄漏
数据库超时
fields @timestamp, @message
| filter @message like /database/ and @message like /timeout|error|failed/
| parse @message "timeout after * seconds" as timeout_duration
| stats count() as timeout_count,
avg(timeout_duration) as avg_timeout
by bin(5m)
内存泄漏检测
fields @timestamp, memory_used_mb, heap_size_mb
| stats avg(memory_used_mb) as avg_memory,
max(memory_used_mb) as max_memory
by bin(1h)
| sort @timestamp asc
场景 6：请求追踪
fields @timestamp, @message, request_id, service, action
| filter request_id = "abc123def456"
| sort @timestamp asc
多服务追踪
fields @timestamp, service, action, duration_ms, status
| filter request_id = "abc123def456"
| sort @timestamp asc
| display @timestamp, service, action, duration_ms, status
高级技巧：parse 与正则表达式
解析非结构化日志
fields @timestamp, @message
| parse @message "[*] User * failed to access resource * from IP *"
as level, user, resource, ip
正则解析
| parse @message /Request completed in (?<duration>\d+)ms with status (?<status>\d+)/
高级技巧：计算字段
fields @timestamp, requests, errors
| fields error_rate = (errors / requests) * 100
fields @timestamp, status_code
| fields status_category =
case(status_code < 300, "success",
status_code < 500, "client_error",
status_code >= 500, "server_error")
| stats count() as request_count by status_category
查询优化
异常检测与关联
统计异常检测
- 计算基线：7 天的均值与标准差
- 将当前区间与基线对比
- 标记 > 2 个标准差的值为异常
错误率关联分析
fields @timestamp, level
| stats count() as total_requests,
sum(case(level = "ERROR", 1, 0)) as errors
by bin(5m)
| fields error_rate = (errors / total_requests) * 100
总结
- CloudWatch Logs Insights：交互式查询服务，无需基础设施
- 三种查询语言：Logs Insights QL、OpenSearch PPL、OpenSearch SQL
- 六个排障场景：错误激增、端点变慢、认证失败、DB 超时、内存泄漏、请求追踪
- 高级技巧：用于非结构化文本的
parse，用于序列的bin()，用于分位数的pct() - 关联与异常检测：错误率分析、统计基线对比
- 从排查到自动化：指标筛选与告警闭环
让我们一起练习吧！
AWS 监控与故障排查
