Firefly

The full case study lives at try-firefly.github.io. This page is the short version.

What it is

Firefly is an open-source observability framework for serverless functions. It solves a specific problem: serverless workloads are stateless and ephemeral, so the usual “install an agent on the box” approach to observability doesn’t work, and the SaaS replacements get expensive fast (vendor pricing can hit ~50% of infrastructure cost).

Firefly is the middle option: self-hosted, automated to deploy, vendor-neutral.

Architecture

Three phases:

1. Emit. AWS Distro for OpenTelemetry (ADOT) Lambda layer auto-instruments functions for traces. Metrics are pushed via CloudWatch Metric Stream → Kinesis Data Firehose, so data leaves the function only when there’s data to send (no pull-based polling latency).
2. Ship. A custom OpenTelemetry collector gateway accepts traces and metrics, transforms metrics from CloudWatch JSON into Prometheus format, and writes to a single backend.
3. Present. Promscale (built on TimescaleDB) stores both traces and metrics; Grafana renders pre-built dashboards over Prometheus, Jaeger, and Postgres query interfaces.

What I worked on

• Context propagation across event-based Lambda invocations: reconfigured ADOT to use the W3C TraceContext protocol instead of AWS X-Ray headers, and built a wrapper layer that intercepts requests to extract traceparent headers from non-standard locations.
• A custom wrapper for AWS SDK Lambda invocations that injects traceparent into the invocation payload: the missing piece for tracing across function boundaries.
• Database selection: pushed for Promscale over Amazon Timestream after benchmarking (5-175x faster reads, ~6000x higher insert throughput, ~220x cheaper for our workload, and cloud-agnostic).

What we explicitly didn’t build

• Logs. Too granular for the framework’s value proposition; left to existing CloudWatch tooling.
• Non-Node.js or non-AWS support. Future scope, not v1.

Team

Built with Julia Park (Toronto), Marcus Sinkinson (Amsterdam), and Carolina Avila (NY) as a Launch School capstone project. The case study credits the team collectively; specifics on who-owned-what are in the writeup linked above.