Not working everywhere, is there anything wrong?

[JounQin]

// test.mjs
import { createWorker } from 'await-sync';

const awaitSync = createWorker();

const sync = awaitSync(async () => {
  return new Uint8Array();
});

console.log(sync());

Node.js:

node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_WORKER_INVALID_EXEC_ARGV]: Initiated Worker with invalid execArgv flags: --import
    at new NodeError (node:internal/errors:399:5)
    at new Worker (node:internal/worker:202:13)
    at #loadModule (file:///Users/JounQin/Workspaces/Study/worker-study/node_modules/whatwg-worker/node-worker.js:96:20)
    at new Worker (file:///Users/JounQin/Workspaces/Study/worker-study/node_modules/whatwg-worker/node-worker.js:49:21)
    at createWorker (file:///Users/JounQin/Workspaces/Study/worker-study/node_modules/await-sync/mod.js:14:18)
    at file:///Users/JounQin/Workspaces/Study/worker-study/test.mjs:3:19 {
  code: 'ERR_WORKER_INVALID_EXEC_ARGV'
}

Bun.js:

1 | export default Worker;
                  ^
ReferenceError: Can't find variable: Worker
      at /Users/JounQin/Workspaces/Study/worker-study/node_modules/.deno/[email protected]/node_modules/whatwg-worker/browser-worker.js:1:15

Deno

error: Uncaught (in worker "") TypeError: Could not resolve 'blob:null/a1a3f86f-a252-4652-8491-8eac3194ed14' from 'file:///Users/JounQin/Workspaces/Study/worker-study/node_modules/.deno/[email protected]/node_modules/await-sync/worker.js'.
  Caused by:
    [ERR_UNSUPPORTED_ESM_URL_SCHEME] Only file and data URLS are supported by the default ESM loader. Received protocol 'blob'
    at async file:///Users/JounQin/Workspaces/Study/worker-study/node_modules/.deno/[email protected]/node_modules/await-sync/worker.js:29:27

[jimmywarting]

Hmm, darn...
i tested this in NodeJS v19 and it worked out just fine then. and now when i went back to test it out in a newer version (v20) it didn't work anymore...

To answer your issues:

Node.js:

Initiated Worker with invalid execArgv flags: --import
it seems like the NodeJS version you tested this on are a bit older, as it didn't have support for it.
to help out you better i would need to know which NodeJS version you use.

Bun.js:

Hmm, seems like there are bigger issues here...
deno don't have postMessage, MessageChannel, BroadcastChannel, web worker or even support for NodeJS worker_threads

so this package will likely not even work in bun.js until they implement something

i have never actually really tested out bun.js before until today.
i just assumed that it had support for web Workers

deno

hmm, 🤔
i remember having seen the error ERR_UNSUPPORTED_ESM_URL_SCHEME before in a unrelated project
and it's coming from NodeJS. and it sure was coming from NodeJS, so i don't know how you got this error message.

i have actually tested this package out in deno by running:deno run test/spec.js and it runs just fine.
i have tested loading this via some CDN... some fails by trying to mess around with trying to convert every package to esm.

• jspm fails, created an issue for it here: it messes with import.meta.url jspm/project#341
• jsdeliver don't even list my new package, but the cdn works regardless from https://cdn.jsdelivr.net/npm/await-sync/mod.js just using https://cdn.jsdelivr.net/npm/await-sync cuz of the same reason as jspm. i think it would be better if it tried to redirect to the main file instead.
• esm.sh dose actually a good job correctly.

so maybe try this out in deno:

import { createWorker } from 'https://esm.sh/[email protected]/mod.js'

onst awaitSync = createWorker();

const sync = awaitSync(async () => {
  return new Uint8Array();
});

console.log(sync());

[JounQin]

it seems like the NodeJS version you tested this on are a bit older, as it didn't have support for it.
to help out you better i would need to know which NodeJS version you use.

I'm using v18.16.0

---

On Node 19, it works a bit unexpectedly:

import { createWorker } from 'await-sync'

const awaitSync = createWorker();

const sync = awaitSync(async () => {
  console.log('xxxx')
  return new Uint8Array([0, 1]);
});

console.log(sync());

Uint8Array(2) [ 0, 1 ]
xxxx  // shouldn't it be logged first?

---

For Deno, import { createWorker } from 'https://esm.sh/[email protected]/mod.js' this works to me now, perhaps import { createWorker } from 'await-sync' runs in node compat mode. 😂

And Deno runs in the expected sequence:

xxxx
Uint8Array(2) [ 0, 1 ]

[jimmywarting]

On Node 19, it works a bit unexpectedly:

Hmm 🤔
That is weird. i guess what happens is:
that console logs in worker_threads aren't really logged to the terminal in the workers. I suppose the main threads just reads the stdout from another process and then main thread just pipe the worker.stdout via a stream to the main process. and that streams uses some next event loop stuff internally. so:

• the main threads creates a worker and pipes the worker.stdout to it's own process.stdout in the main thread
• the main thread freezes (gets blocked). so nothing can ever be logged.
  i notice this when i tried to debug/log stuff when the main thread was frozen and got nothing back. had to use the file reader to dump logs to a file instead...
• the main thread unfreeze and continue running console.log(sync()) where it left of.
• then process.stdout.write() gets called afterwards

---

I'm going to put in some investigation into trying to get it to work on older and newer nodejs versions.
it kind of is in beta stage ATM. haven't really had much time testing this package out much yet. it's relative new.

I wanted my whatwg-worker to support importing stuff from blob urls and https imports too. it was a little side-dependency/useful tool i wanted to have along with this project so both are relative new both are not even 3 weeks old.

So excuse me if things are not working properly.

---

To support importing from other sources other than npm i had to add in experimental loaders flag when starting up worker_threads.Worker with custom execArgs but i guess they are very experimental ATM. the http loader have changed quite much between all releases. and now it apparently run in a total other context too that broke some stuff in v20

The reason for doing so was so that i could import code that is dynamically created.
kind of goes like

blob = new Blob([code])
url = URL.createObjectURL(blob)
import(url)

And import('blob:<uuid>') isn't quite yet supported today in NodeJS
there is an issue for this here: nodejs/node#47573
and this is why i had to use experimental loaders to make it work.
give that issue a 👍 if you want to see less hacky stuff in this package.

[jimmywarting]

I may have found a way to improve the logging while the main thread is blocking.

i found out how NodeJS do it here: https://github.com/nodejs/node/blob/0b3fcfcf351fba9f29234976eeec4afb09ae2cc0/lib/internal/worker/io.js#L360-L396

It basically did confirmed what i tough and said. they send the logs to the parent thread instead with postMessages.

An alternative solution would be to take the process.stdout.fd from the main thread and write directly to that one instead in the worker instead of doing postMessage stuff

I'm experimenting with it now. to be able to console log stuff while the main thread is blocked

Edit
Seems to work out okey

[jimmywarting]

Hmm, i guess you have used a package.json when running deno?

i have had this in the package.json

  "exports": {
    "browser": "./browser-worker.js",
    "node": "./node-worker.js"
  },

do you think that is the reason why?
do deno have own support for this?
would it work if i added

  "exports": {
    "browser": "./browser-worker.js",
    "deno": "./browser-worker.js",
    "node": "./node-worker.js"
  },

[jimmywarting]

1.0.1 release 🎉

fixes console logs so they logs in the correct order
And you can now also use console.log even dough the main thread is blocked, which helps a LOT!
now instead of using postMessages it writes directly to the stdio file descriptor.

so

• fix console log order + debugging while main thread is blocked
• improve better NodeJS compatibility in other versions
• making sure that Deno dose not use NodeJS own runtime when installing from package.json