pg-boss use case: queue everything

[willypt]

Firstly, thank you @timgit for creating pg-boss. This is by far my favorite queueing system to date!
The system is reliable and can be deployed + integrated very easily 👏🏻

I'm currently using pg-boss to ensure I can queue all my jobs respecting singleton capability. When firing 8 subsequent jobs (2 key A and 2 key B), it seems I'm only able to queue at max 4 jobs (1 completed, 1 created only for each key)

async function readme() {
    const PgBoss = require('pg-boss');
    const boss = new PgBoss('postgres://willypt:[email protected]/pgboss');

    boss.on('error', error => console.error(error));

    await boss.start();

    const queue = 'some-queue-1';

    async function sendJobA() {
        return await boss.send(queue, {
            params: {
                timestamp: new Date().getTime()
            },
            eventMetadata: 'exec-metadata'
        }, {
            singletonKey: '123456',
            singletonSeconds: 60,
            singletonNextSlot: true,
        })
    }

    let jobId1 = await sendJobA()
    let jobId2 = await sendJobA()
    let jobId3 = await sendJobA()
    let jobId4 = await sendJobA()

    async function sendJobB() {
        return await boss.send(queue, {
            params: {
                timestamp: new Date().getTime()
            },
            eventMetadata: 'exec-metadata'
        }, {
            singletonKey: '7890123',
            singletonSeconds: 60,
            singletonNextSlot: true,
        })
    }

    let jobId1b = await sendJobB()
    let jobId2b = await sendJobB()
    let jobId3b = await sendJobB()
    let jobId4b = await sendJobB()

    console.log(`created job in queue ${queue}: ${jobId1}, ${jobId2}, ${jobId3}, ${jobId4}`);
    console.log(`created job in queue ${queue}: ${jobId1b}, ${jobId2b}, ${jobId3b}, ${jobId4b}`);

    await boss.work(queue, someAsyncJobHandler);
}

async function someAsyncJobHandler(job) {
    console.log(`job ${job.id} received with data:`);
    console.log(JSON.stringify(job.data));
}


readme();

This solves into 2 jobs queued

created job in queue some-queue-1: ad38723f-21f2-4089-8441-43439ac8d54a, 097bd23e-d54e-4bd5-ba56-a4cdcd41e4c8, null, null
created job in queue some-queue-1: 70f4177a-719d-424a-af1c-8cbe29ad4dff, ffd8a920-405d-40f0-9576-c1657d945ddb, null, null

The expected is that I should be able to create four tasks and queue each of them per their singleton slot

Wondering if this is achievable with the current pg-boss and would you be open for PR and if this is something you'd like to see too @timgit ?

[willypt]

@timgit Interesting, I just realized that you're working on v10 which seemed to be "resolving" my current ask via createQueue. May I know if the concept of it is currently available in v9? 🤔

[timgit]

@willypt, thanks, and it's good to hear it's working for you.

createQueue() will not be in v9. v10 is a much larger change to the API and infrastructure, so it's taking a bit longer as I weigh the pros/cons of release and test migration scenarios.

[willypt]

Well understood. Thanks for clarifying @timgit . Looking forward to the next version being official!

Otherwise, for those stuck at v9 and would like to get a bit of preview, I've tried to do some tinkering with v9 to add support for sendQueued() that should simulate createQueue(singleton) in v10 🎉
Please exercise testing before using this in your production environment.

The PR is closed favoring the better support for v10

#435