Unable to reach quorum

[paymog]

I'm having some trouble using this library. I have an endpoint which gets called twice upon page load and I'm trying to use a lock to perform sensitive database operations safely.

I'm using this library and I keep getting the following error in unusual ways which I haven't been able to figure out (maybe I have the library misconfigured?). Here's the code I'm using to initialize redlock:

import { getRedisURI } from './settings/settings.js';
import Redis from 'ioredis';
import Redlock from 'redlock';

export const redis = new Redis(getRedisURI());
export const redlock = new Redlock([redis]);

here's the code which does the sensitive database work:

async function createNewScheduleCardsIfNecessary(tripId) {
  let lock;
  console.log('attempting to make new schedule card items for trip', tripId);

  try {
    lock = await redlock.acquire([`${tripId}`], 5000, {
      retryDelay: 1000,
      retryCount: 30,
      automaticExtensionThreshold: 100,
      retryJitter: 1000,
    });
    console.log('acquired lock', tripId);
    const expectedNumberOfCards = await getExpectedNumberOfScheduleCards(tripId);
    const currentCards = await ScheduleCards.find({
      tripId,
    }).toArray();

    await createMissingScheduleCards(currentCards, expectedNumberOfCards, tripId);
    console.log('finished making schedule cards', tripId);
  } finally {
    if (lock) {
      console.log('releasing lock', tripId);
      await lock.release();
    }
  }
  console.log('all redlock stuff finished for trip', tripId);
}

here's the full output I'm getting:

Server running at http://localhost:5001/graphql


schedule cards called for trip vMEKsgzFy9EGdQKbp <------------ this indicates a request came in from the web 
attempting to make new schedule card items for trip vMEKsgzFy9EGdQKbp
acquired lock vMEKsgzFy9EGdQKbp
schedule cards called for trip vMEKsgzFy9EGdQKbp <--------------- here's the second request from the web
attempting to make new schedule card items for trip vMEKsgzFy9EGdQKbp
need to create schedule card for trip { part: 'morning', i: 0 }
need to create schedule card for trip { part: 'morning', i: 1 }
need to create schedule card for trip { part: 'morning', i: 2 }
need to create schedule card for trip { part: 'morning', i: 3 }
need to create schedule card for trip { part: 'morning', i: 4 }
need to create schedule card for trip { part: 'afternoon', i: 0 }
need to create schedule card for trip { part: 'afternoon', i: 1 }
need to create schedule card for trip { part: 'afternoon', i: 2 }
need to create schedule card for trip { part: 'afternoon', i: 3 }
need to create schedule card for trip { part: 'afternoon', i: 4 }
need to create schedule card for trip { part: 'evening', i: 0 }
need to create schedule card for trip { part: 'evening', i: 1 }
need to create schedule card for trip { part: 'evening', i: 2 }
need to create schedule card for trip { part: 'evening', i: 3 }
need to create schedule card for trip { part: 'evening', i: 4 }
need to create schedule card for trip { part: 'lodging', i: 0 }
need to create schedule card for trip { part: 'lodging', i: 1 }
need to create schedule card for trip { part: 'lodging', i: 2 }
need to create schedule card for trip { part: 'lodging', i: 3 }
need to create schedule card for trip { part: 'lodging', i: 4 }
finished making schedule cards vMEKsgzFy9EGdQKbp
releasing lock vMEKsgzFy9EGdQKbp
ExecutionError: The operation was unable to achieve a quorum during its retry window.
    at Redlock._execute (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/node_modules/redlock/dist/esm/index.js:290:23)
    at async createNewScheduleCardsIfNecessary (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/api/scheduleCards/graphql/queries.js:143:7)
    at async Object.scheduleCards (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/api/scheduleCards/graphql/queries.js:153:5) {
  attempts: [
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] },
    Promise { [Object] }
  ]
}
ExecutionError: The operation was unable to achieve a quorum during its retry window.
    at Redlock._execute (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/node_modules/redlock/dist/esm/index.js:290:23)
    at async Redlock.acquire (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/node_modules/redlock/dist/esm/index.js:207:34)
    at async createNewScheduleCardsIfNecessary (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/api/scheduleCards/graphql/queries.js:126:12)
    at async Object.scheduleCards (file:///Users/paymahn/code/tripvector/tripvector-mono/backend/api/scheduleCards/graphql/queries.js:153:5) {
  attempts: [
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }, Promise { [Object] },
    Promise { [Object] }
  ]
}

The time between the acquired lock and releasing lock log lines is around 1 second but for some reason, even with the generous retry policy I'm seeing redlock fail consistently.

The top stack trace includes line 143 which is the line where I release the lock. The bottom stack trace include line 126 which is where I acquire the lock.

I've tried using redlock.using as well but had the same results. I've downloaded a redis gui to inspect my local redis instance to see if there are any spurious keys but haven't been able to find any.

Here's my package.json:

    "@babel/runtime": "^7.16.3",
    "@graphql-tools/schema": "^8.3.1",
    "apollo-server-express": "^3.5.0",
    "axios": "^0.26.1",
    "bcryptjs": "^2.4.3",
    "body-parser": "^1.19.0",
    "chalk": "^4.1.2",
    "compression": "^1.7.4",
    "cookie-parser": "^1.4.6",
    "cors": "^2.8.5",
    "crypto-extra": "^1.0.1",
    "dayjs": "^1.10.7",
    "ejs": "^3.1.6",
    "express": "^4.17.1",
    "graphql": "^16.0.1",
    "graphql-redis-subscriptions": "^2.4.2",
    "graphql-subscriptions": "^2.0.0",
    "graphql-ws": "^5.7.0",
    "html-to-text": "^8.1.0",
    "immutability-helper": "^3.1.1",
    "ioredis": "^5.0.4",
    "jsonwebtoken": "^8.5.1",
    "juice": "^8.0.0",
    "lodash-es": "^4.17.21",
    "moment": "^2.29.3",
    "mongo-uri-tool": "^1.0.1",
    "mongodb": "^4.2.0",
    "nodemailer": "^6.7.2",
    "ps-node": "^0.1.6",
    "redlock": "^5.0.0-beta.2",
    "serve-favicon": "^2.5.0",
    "speakingurl": "^14.0.1",
    "ws": "^8.5.0"

[paymog]

Downgrading to version 4 for redlock and ioredis solved my issue completely, I guess there's a bug with the v5 implementation.

[kylecannon]

having this same issue too.

[jskorlol]

I have the same problem, is there any solution?

[paymog]

@jskorlol try downgrading to version 4 of this package and ioredis

[kylecannon]

yeah, the problem is i would really like the lock auto extend functionality.

[jjm340]

I'm having this problem too, we use redlock v4 to great success so I'm downgrading for now.

[hlongvu]

any update on this, I am having this problem too

[phil-r]

Hey, we also encountered this issue on both v4 and v5.
In our case the problem was, that our lock was expired by the time we tried to release.

Failing code:

// redis and redlock setup is omitted

const wait = async (ms: number) => {
  return new Promise((resolve) => setTimeout(() => resolve(ms), ms));
};

const main = async () => {
  try {
    const lock = await redlock.acquire(['a'], 1000);
    await wait(1500);
    await lock.release();
  } catch (e) {
    console.error(e);
    throw e;
  }
};

main();

Error:

ExecutionError: The operation was unable to achieve a quorum during its retry window.

working code:

// redis, redlock and wait setup is omitted

const main = async () => {
  try {
    const lock = await redlock.acquire(['a'], 1000);
    await lock.extend(1600);
    await wait(1500);
    await lock.release();
  } catch (e) {
    console.error(e);
    throw e;
  }
};

main();

for us switching to redlock.using, increasing lock duration or using lock.extend all worked.

I think the best for the library would be to make lock.release() a noop if it's already expired.

[Dakuan]

We are still seeing this intermittently even with redlock.using

[jketcham]

I was also running into this on version 5.0.0-beta.2 while using redlock.using, here's the relevant stack trace:

ExecutionError: The operation was unable to achieve a quorum during its retry window.
    at Redlock._execute (file:///workspace/node_modules/redlock/dist/esm/index.js:290:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Redlock.acquire (file:///workspace/node_modules/redlock/dist/esm/index.js:207:34)
    at async Redlock.using (file:///workspace/node_modules/redlock/dist/esm/index.js:448:20)
    ... my code ...

With the call to using looking like:

await redlock.using([lockId], 5000, async (signal) => { ... my code ... });

The problem seems to occur when the same resource is trying to be locked multiple times in close succession. Increasing the retryCount and retryDelay (to 30 and 1000 respectively for my setup) prevented the quorum error from throwing, but then I'd still have errors about not being able to extend an already expired lock.

[eth-limo]

Got bitten by this as well

    "ioredis": "^5.2.3",
    "redlock": "^5.0.0-beta.2",

[paulohlips]

Hi folks, I faced this problem, but was a mistake in my code.

I was trying to aquire and make a set in the same key, as:

const main = async () => {
try {
const lock = await redlock.acquire(['myKey], 1000);
redisClient.set("myKey", "newValue")
await lock.release();
} catch (e) {
console.error(e);
throw e;
}
}

But this way is wrong because we need to aquire a resourceKey like "myLokedResouce:myKey" and then make changes in our key (is tihis exemple "myKey") and after release the lock on "myLokedResouce:myKey". Now my code search by "resourceKey" to know if can makes changes in my "key" like:

const key = "project"
const value = "CONFLICT"
const resource = locks:${key}
const lockTtl = 66000

async function lockAndSet(resource , lockTtl) {
const lock = await redlock.acquire([resource], lockTtl)

try {
await redisSet(key, value)
console.log('Time finished, key unlocked!')
await lock.release()

} catch (e) {
console.log(e)
}
}

lockAndSet(resource, lockTtl)

[JeromeFire]

I have issue too! I have no idea. I read code and find client.evalsha(args) function throw error Error: node_redis: The EVALSHA command contains a invalid argument type.,then I have no idea