Problem
The efficiency report shows 40% drop rate. The @aws-sdk/client-eventbridge stream processor is choking on raw event streams. We’ve got a batch size of 50 configured in the consumer. The conversation:details events drop when source_id has an underscore like cust_ext. The regex filter is too strict for the new schema evolution.
const filter = /source_id.*^cust_\d+/;
// Circuit breaker resets after 5s
if (!filter.test(event.detail.source_id)) {
stats.dropped++;
return;
}
batch.push(event);
Error
It’s logging dropping for cust_ext_99. The circuit breaker trips after 3 consecutive drops and kills the consumer thread. Schema validation fails on the schema_version field because the payload structure shifted.
Question
Need a way to relax the regex without losing the batch aggregation logic. How to route events to multiple consumers based on the topic pattern? Schema v2 breaks the filter.
Updating the regex to allow underscores fixes the pipeline. Clean source data’s essential for seasonal ARIMA modeling. You’ll need to normalize those identifiers before they hit the forecasting queue. Are you tracking schema drift in your historical datasets? The WFM data cleaning post covers similar issues. Honestly, data drift breaks the curves. Check the retry logic.
The regex tweak actually fixes the drop rate. Tested it against our inbound routing scripts and the circuit breaker stopped tripping. The documentation for the EventBridge rule engine states: “Pattern matching applies to the entire string and rejects any character outside the specified class.” So when cust_ext hits the filter, it bombs out because the underscore isn’t in the default alphanumerics. You’ll need to push that updated pattern into the Studio REST proxy action too. If a SNIPPET maps the source_id before it hits the external webhook, the ASSIGN action expects a clean string. Here is how we handle it in the proxy payload:
The docs say the payload must match the schema exactly, but why does the platform still choke on valid underscores when the regex is corrected at the node level. Validation seems to run twice. You’ll have to wrap the identifier in a trim function inside an IF/ELSE block to strip trailing whitespace that the parser might misread. Running the test suite now shows zero drops. Don’t forget to add the event: scope to the service account or the REST proxy will just 401. You can verify the format directly with platformClient.RoutingApi.getRoutingAgent(agentId) before it hits the bridge, or just hit /api/v2/routing/agents/{agentId} via curl to see how the platform normalizes underscores. Still waiting on a patch for the parser.
The regex update works. My Django analytics pipeline was dropping records because Celery tasks rejected the cust_ext format before hitting the /api/v2/analytics/conversations/details/queries endpoint. The workers don’t choke anymore.
Code
Updated the validation in the task handler. Added underscore support to the pattern.
import re
from celery import shared_task
SOURCE_ID_PATTERN = re.compile(r'^[a-z0-9_]+$', re.IGNORECASE)
@shared_task(bind=True, max_retries=3)
def process_source_id(self, source_id):
if not SOURCE_ID_PATTERN.match(source_id):
raise self.retry(countdown=60, exc=ValueError(f"Invalid format: {source_id}"))
# Push to Postgres analytics table
return True
Error
Without this change, the worker throws a DataError on the Postgres insert. The retry loop spins until the broker fills up. It’s a pain to debug in production. The regex tweak stops the churn.
Question
Are you seeing similar drift in the outbound metrics? The bulk limits might choke on the normalized keys. Watch the queue depth.
The regex adjustment resolves the filter issue, though the circuit breaker remains aggressive. In CIC we used to encounter similar rejection patterns when the Attendant configuration tried parsing non-standard extension formats. The EventBridge rule engine treats character class violations as hard failures. Downstream protection triggers instantly. A secondary validation step in the flow stabilizes throughput before the data reaches the endpoint.
Implement this pre-check logic in the Studio script to catch malformed IDs early. The configuration panel requires manual input for custom patterns.