Sentiment Pulse - What Went Wrong (and How I Fixed It)
SentimentPulse started as a simple AWS portfolio project, but it quickly turned into a good lesson in how cloud cost control can go sideways if automation is left unchecked.
This page covers the main problems I ran into while building the project, how I traced them down, and what I changed to make the architecture safer and more predictable.
Unexpected Comprehend Spend from Repeated Batch Runs
The biggest issue was not a broken API call or an obvious application crash. It was repeated execution. I noticed Amazon Comprehend DetectSentiment charges rising in my bill even though the Lambda log group tied to the real-time path was empty.
After tracing the project through Cost Explorer, CloudWatch Logs, ECS log streams, and EventBridge, I found the real cause: an hourly EventBridge rule was launching the SentimentPulse ECS batch task over and over. Each run only processed a small bounded set of records, but the repeated runs added up and created real cost.
In CloudWatch Logs, each batch run showed the same pattern:
- 25 rows processed per run
- approximately 248 billed text units per run
- many separate ECS log streams over time
Individually, those runs looked harmless. Collectively, they explained the bill.
Fix: I removed the always-on schedule and changed the Terraform design so scheduled execution is optional and disabled by default. The batch path is now something I enable intentionally, not something that quietly runs in the background.
Detecting the Wrong Suspect at First
One reason this took time to diagnose was that I initially focused on Lambda. Since the public demo includes a real-time API path, that was the obvious first place to look. But the Lambda log group was empty, which forced me to widen the investigation.
The breakthrough came from checking the ECS log group instead of relying on a single Lambda log stream. Once I saw repeated batch-start and batch-complete messages, the pattern became clear: the ECS task was the source of the Comprehend activity, not Lambda.
Lesson learned: when investigating AWS costs, logs, metrics, scheduler rules, and billing views all need to be correlated. Looking in only one place can send you down the wrong path.
Image Tag Drift and \":latest\" Problems
I also discovered that the ECS task definition was not always running the exact image I thought it was. Earlier versions of the project relied too heavily on mutable image tags, which made it easier for older container versions to stick around longer than expected.
Fix: I moved to a digest-pinned ECR image reference for the task definition. That way ECS runs the exact image I built and tested, rather than whatever a floating tag happens to point at.
Terraform Cloud Variable Precedence
Another subtle issue came from Terraform Cloud workspace variables. I had a value defined in code, but the workspace still had an older image URI set. Since Terraform Cloud workspace variables take precedence, my changes in code were being silently overridden.
Fix: I removed the ambiguous fallback behavior and now treat the image URI as an explicit required input. If the workspace value is wrong, the deployment should fail loudly instead of quietly using an old image.
Batch Safety Controls
After seeing how easily repeated runs can add up in a metered AI workflow, I added stronger guardrails to the batch path:
- MAX_RECORDS cap so each run only processes a small bounded number of records
- Digest-pinned container image so the task runs the exact tested version
- Optional scheduler only so automatic execution is no longer the default
- AWS Budget alerts to surface spend earlier
- CloudWatch log review for run counts, billed-unit estimates, and troubleshooting
The point was not just to stop one mistake. It was to redesign the project so the same class of mistake is less likely to happen again.
Monitoring and Cost Awareness
This project taught me that debugging AWS cost is its own skill. Cost Explorer, CloudTrail, CloudWatch Logs, ECS, and EventBridge all showed different parts of the story. None of them, by themselves, told the whole truth.
Fix: I now treat cost visibility as part of the architecture. Budget alerts, bounded processing, safe Terraform defaults, and log-based verification are all part of keeping a public demo under control.
Outcome
SentimentPulse is in a much better state now. The architecture still demonstrates ECS Fargate, Comprehend, S3, Lambda, API Gateway, and Terraform, but the batch path is now safer and more intentional.
The painful part of this incident is also what made it valuable: it forced me to think about cost control the same way I think about security and reliability. Good cloud architecture is not just about getting a service to run. It is about making sure it behaves safely after you walk away from it.