Flush Race Condition

[thejuan]

I'm using segment inside an AWS Lambda.
At the moment we identify and track throughout the lambda then call flush at the end and wait for it to complete

When flushAt is set to 1, and we make a single identify call then we wait for flush to complete before letting the lambda finish (we thought so that all inflight requests could complete).

However flush doesn't seem to track the previous inflight flushes called internally and it returns immediately, (as in this case the queue is empty) even if a previous flush is still in progress.

If we set flushAt = 2 and make 2 track calls the same thing will happen.

Are we meant to track all calls to identify and track and wait on them manually?

[SilverSting]

Hi @thejuan

I had a similar experience. What I've found at the end was that the very first tracking call is flushed straight away.

Take a look at the if (!this.flushed) condition inside the enqueue method, line 206 of index.js. As the queue has never been used, the flushed state is false, and your very first tracking call will be flushed straight away. And because flushing is asynchronous, it may give you unexpected behaviour, like the follow up flush call giving you incorrect queue status.

    this.queue.push({ message, callback })

    if (!this.flushed) {
      this.flushed = true
      this.flush()
      return
    }

I ended up doing something like this below for my AWS Lambda function.

    // IMPORTANT, segment flushes the very first call, wait for the flush callback.
    // For an added safety, disable the queue by setting { flushAt: 1 }
    segment.flushAt = 1
    await (function () {
      return new Promise((resolve) => {
        segment.track(payload, () => {
          resolve(true)
        })
      })
    })()

[jasan-s]

@SilverSting could you also just call the following after calling identify and track ?
analytics.flush(function(err, batch){ console.log('Flushed, and now this program can exit!'); });

[pbassut]

Hey everybody, the wait has finished.
You can know await for a flush.
Please understand that your lambda function will halt in order to wait for the flush to happen, but that's implicit.

Please take a look at the newest release and make sure that fixes this issue so we can close it for good.

[timkinnane]

@pbassut I found this issue and at first it wasn't making sense because the @types/analytics-node package has typed the flush method to return the Analytics class instance, not a promise.

Looking into the actual module though, it does appear to return a promise. The types are just out of sync with implementation.

Types:

https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/analytics-node/index.d.ts#L100

Source:

https://github.com/segmentio/analytics-node/blob/master/index.js#L288

[TigerWolf]

I stumbled across this as I was having this problem in a lambda.

await analytics.flush(function (err, batch) {
  console.log('Flushed, and now this program can exit!', err, batch);
});

Did not work.

I added a
await new Promise(resolve => setTimeout(resolve, 500));
and it started working.

By implementing @SilverSting solution I have now got it working without a setTimeout.

Is there something wrong with the flush method? Should I create a new issue?

[silesky]

@TigerWolf can I invite you to try the private beta of our new node SDK? The instructions for usage in lambda are here:
https://github.com/segmentio/analytics-next/tree/master/packages/node#usage-in-aws-lambda

[TigerWolf]

I tried using the version you linked but I had more errors. I ended up implementing a similar solution to the docs but using the old version.