Efficiently advancing simulated AWS time without real timers
Yulin has a simulated clock feature that lets you freeze, resume and advance time in each simulated AWS instance. Advancing the sim clock triggers any scheduled tasks like EventBridge schedules that would have occurred in that interval before returning.
The first implementation of this turned out to have a performance bug which caused it to take around 17 seconds to advance the sim clock by 30 days. That might still sound like quite a good ratio, but it should take milliseconds. One of the main features of Yulin is the ability to test significant system behaviours in a small amount of real time.
A relatively short example showing the kind of situation where the performance problem would arise:
import { PutMetricAlarmCommand } from "@aws-sdk/client-cloudwatch";import { SimAws } from "@kensio/yulin";
const simAws = new SimAws();const metrics = simAws.cloudWatch();
for (const AlarmName of ["OrdersFailing", "PaymentsFailing"]) { await metrics.putMetricAlarm( new PutMetricAlarmCommand({ AlarmName, Namespace: "Orders", MetricName: "Failed", Statistic: "Sum", Period: 300, EvaluationPeriods: 1, DatapointsToAlarm: 1, Threshold: 5, ComparisonOperator: "GreaterThanThreshold", }), );}
await simAws.clock().advanceBy({ days: 30 });Advancing the sim clock does not skip straight to the end. It runs any events and tasks that would
have occurred in that interval before the advanceBy() call returns. This all happens inside the
same single Node.js process, so you can do things like collecting test coverage and stepping through
the system in a debugger.
Because of that need, any work scheduled between the starting sim time and the end of the interval has to be triggered. The implementation steps through each instant at which something is due to occur. Yulin currently evaluates each sim CloudWatch alarm at its configured period boundary.
Thirty days of five-minute intervals is 8,640, and with the two sim CloudWatch alarms configured in the example above, that doubles to 17,280.
Computers are quite fast, so 17,280 in itself is not a large number for a computer to cope with. The performance problem arose because the first implementation of this used a Node.js timer to defer background tasks during the interval.
The scheduler passed a zero-millisecond delay, equivalent to setTimeout (..., 0). Node clamps any
timeout below one millisecond to one millisecond, but also does not guarantee that it will run at
exactly the scheduled time. That meant that every trip through that path introduced a real timer
with a minimum requested delay of one millisecond.
Those accumulated one-millisecond waits were unnecessary anyway. The first implementation of the
advanceBy() loop already waited for each task to finish before it picked up the next one, so
deferring tasks with a timer was just shuffling work around on the loop without fundamentally
changing anything.
The fix was pretty small:
private async runDueTasksUpTo(instant: Date): Promise<void> { let due = this.background.takeNextDueBy(instant);
while (due !== undefined) { this.clock.setTo(this.laterOfNow(due.dueTime));
this.background.schedule(due.task); await due.task();
await this.background.complete();
due = this.background.takeNextDueBy(instant); }}The performance fix was to replace this.background.schedule(due.task); with await due.task();.
That avoided the problem with minimum one-millisecond waits on setTimeout(0).
After that change, advancing the sim clock by 30 days went from around 17 seconds down to around 24 milliseconds, more than a 700-fold improvement.
The main takeaway from this is that Yulin has to keep simulated AWS time separate from real time in the host process. For behaviour driven by simulated AWS time, the host clock should not accidentally introduce real waits. Separating simulation time from real time is what allows for efficient lightweight testing and flexible local development.
Software Engineering by Kensio Software
Documenting Yulin v1.20.16
