Remove object wrappers from API payloads

[benfrancis]

Blocks: #2802

Currently for a property definition in a Thing Description exposed by the gateway as below:

...
    "temperature": {
      "type": "number",
      "links": [{"href": "/things/pi/properties/temperature"}]
    }
...

a response to a GET request on the Property resource looks something like:

{
  "temperature": 21
}

where the payload is a JSON object with a map representation of the property's name and value. The payloads of PUT requests and responses also follow the same format.

To be compliant with the W3C Thing Description specification, let's change payloads to match the data schema in the Thing Description, e.g. for the above example this would simply be:

21

See WebThingsIO/api#143 for some background on this topic.

This includes:

1. Removing object wrappers from HTTP requests and responses on a Property resource when reading and writing properties
2. Removing object wrappers from HTTP requests and responses when invoking an action on an Action resource
3. Updating the gateway's front end to expect this

The excludes:

1. Removing object wrappers from Properties resources (I seem to recall the W3C specification actually defined an object wrapper for readallproperties and writeallproperties ops with values keyed by property names, but I can't find it in the current draft, can anyone confirm this?)
2. Changing the payload of the Things resource, which will hopefully be defined in the WoT Discovery specification as part of the Directory Service API
3. Changing payloads of any messages in the WebSocket API, which need the additional context

Open questions:

1. What should we do about Event resources, which currently return an array of objects in their HTTP response?
2. What should we do about Events resources, which currently don't have an obvious representation which is W3C compliant?
3. What should we do about Action resources which represent an action queue?
4. What should we do about Actions resources?

[benfrancis]

Some thoughts on the open questions...

1. What should we do about Event resources, which currently return an array of objects in their HTTP response?
2. What should we do about Events resources, which currently don't have an obvious representation which is W3C compliant?

I've submitted a proposal to the W3C for how to standardise this using forms w3c/wot-thing-description#892 (comment) but it's not pretty.

If we don't think this is going to get into WoT Thing Description 1.1 we have a couple of options for WebThings Gateway:

1. Remove the event log feature from the REST API and limit event logs in the web interface to only displaying events that were emitted since the user started viewing the log, by opening a WebSocket when they open the log UI
2. Remove the HTTP endpoints from event affordances in the Thing Description but leave the REST API in place, allowing the gateway's web interface to use this proprietary extension of the API using hard-coded paths but not including those paths in the W3C compliant Thing Description

(We would probably also have to remove event log features from all 16 web thing libraries in the WebThings Framework since there would be no W3C compliant way to implement it.)

What should we do about Action resources which represent an action queue?

This has been discussed in the W3C Working Group in w3c/wot-thing-description#302 and w3c/wot-thing-description#899.

I have made a proposal (just slightly tweaked @egekorkan's proposal) in w3c/wot-thing-description#302 (comment)

If we don't think this is going to get into WoT Thing Description 1.1 then we may have to remove the concept of action queues from WebThings (until someone comes up with a solution which fits into the W3C forms model at some point into the future) and only support invoking actions. For the gateway this isn't the end of the world because the built-in web interface doesn't currently provide a cancel action feature anyway, but I don't know whether third party consumers of the gateway's API or the users of the WebThings Framework rely on this feature.

What should we do about Actions resources?

Let's worry about this if the action queue issue above gets resolved.

If there's no way to represent these in a W3C Thing Description then we may have to remove this feature too and re-write the gateway front end to use individual action endpoints rather than the top level actions endpoint when invoking actions.

[benfrancis]

• What should we do about Event resources, which currently return an array of objects in their HTTP response?

• What should we do about Events resources, which currently don't have an obvious representation which is W3C compliant?

I've tried to figure out a way to model these resources as interaction affordances in a W3C Thing Description and discussed potential new readpastevents and readallpastevents operations for the W3C WoT Thing Description specification to make it possible (see w3c/wot-thing-description#892).

The hardest part is how to model the payloads of these resources, which not only include the raw event data, but also metadata like timestamps.

In trying to write text for the specification which explains how to describe these resources in a Thing Description I think I've come to the conclusion that we actually shouldn't try. What makes these resources unusual is that they describe historical stored data, rather than live data being read from or emitted directly from a device. Requiring that devices be able to provide a historical log of events assume they have the storage needed to so, which may not always be the case. In our implementation it's actually the gateway that logs this data and not the devices themselves, so you could argue that the event logs are actually a feature of the gateway rather than an affordance of the device itself.

I'd therefore like to propose that we take a different approach:

API

• /things/{thingName}/events/{eventName} - Change the existing Event resources from logs (returning historical data) to Server Sent Events (SSE) sources (emitting live events as they occur). We previously avoided SSE because it didn't have broad browser support, but that has improved in the last few years.
• /things/{thingName}/events - Use the proposed subscribeallevents and unsubscribeallevents operations to describe a top level Events resource for each device which can be used to subscribe to all events via a single endpoint. This is important because SSE implementations in browsers still have a hard limit of six live event sources per domain.
• /logs - In order to not drop the event logging feature altogether, eventually expose event logs via the logs API instead, completely separate from the WoT API (see below)

UI

• In the short term, change the current event log dialog inside the Things screen to only start logging events (on the client side) when the user opens the dialog, using the new Server Sent Event endpoint for the device (or the existing WebSocket endpoint).
• In the longer term, consider adding event logging to the Logs screen so that you can log events as well as properties, with logs stored to persistent storage. This will require some UI design work, especially since not all event payloads are necessarily numerical. Once we have this feature, we could potentially remove the event log dialog from the Things screen altogether.

The quickest way to unblock this issue might be to initially remove the links/forms from the event affordances altogether and modify the event log UI to only show live events using the existing WebSocket endpoint. But ideally we would replace the existing event endpoints with new ones using Server Sent Events.

@relu91 What do you think?

[relu91]

In trying to write text for the specification which explains how to describe these resources in a Thing Description I think I've come to the conclusion that we actually shouldn't try. What makes these resources unusual is that they describe historical stored data, rather than live data being read from or emitted directly from a device. Requiring that devices be able to provide a historical log of events assume they have the storage needed to so, which may not always be the case. In our implementation it's actually the gateway that logs this data and not the devices themselves, so you could argue that the event logs are actually a feature of the gateway rather than an affordance of the device itself.

I have to admit this feature felt a little bit weird from the beginning. Even if I reconsigned its utility, I have never seen IoT devices that kept track of the fired events. Even in our research experiments, we decided that the event source (i.e., the web thing) should not be in charge of taking a log of its events. Pondering the question: "If a tree falls in a forest and no one is around to hear it, does it make a sound?" 🤣 . So, I would not complain if we remove this feature from our WebThings.

As for the proposed approach, it's a similar architecture that we employed in our WoT based monitoring projects so I am aligned with the API design. We have still tried to expose the /logs endpoint with a WoT interface describing virtual Web Things linked with the events sources, but I agree that it might be too much of a stretch.

The quickest way to unblock this issue might be to initially remove the links/forms from the event affordances altogether and modify the event log UI to only show live events using the existing WebSocket endpoint. But ideally we would replace the existing event endpoints with new ones using Server Sent Events.

I would do it in a single dev sprint since Afforndaces could not have empty forms, so we are forced to fill those with the new Server-Sent Events links. Other than that it a good plan 👍🏻

[benfrancis]

Fixed by #2875, #2871 and #2863.