🧑🏾💻
Overview description of the prep work for the sprint
🔌 WebSockets
Learning Objectives
WebSockets are an API and protocol which create a two-way communication channel between two programs.
Websockets are used to establish a channel so that a backend can send updates to a frontend website.
You can read an introduction to WebSockets, as well as roughly what a client looks like, and what a server does.
On the server side, we will be using the websocket npm package which lists a server example in its README.
💡Tip
This sprint, you will need to submit both a copy of your code which supports polling, and a copy which supports WebSockets.
You probably want to make a copy of your polling code, and have two separate (similar) pages in your repo.
Note that this is different from other work you’ve done, where you make different branches for different pieces of work. We want both of your implementations on one branch.
You should aim to share code between these implementations where possible.
On the backend, create a WebSocket server by adding this code:
import { server as WebSocketServer } from "websocket";
const server = http.createServer(app);
const webSocketServer = new WebSocketServer({ httpServer: server });
Next, follow the example in the websocket npm package’s documentation to have your server handle requests.
On the client-side, you will need to make a new WebSocket connection to the server.
Some things to think about when implementing WebSockets updates:
Learn new APIs in isolation
It will be easier for you to learn a new API (like WebSockets) with a simple example.
Your goal is to make a WebSocket to stream new messages from the server to the client. Your existing codebase is more complicated and handles many other concerns. To explore WebSockets, configure the server to always report the message “Hello”. This way, you can isolate the new process and test things out more easily.
You could even write a whole new website which only makes a WebSocket connection and displays a message.
Once you have an example WebSocket working, and understand how it works, keep going. Apply your new understanding to the real problem you’re trying to solve.
Think about the protocol you want
WebSockets let you send arbitrary text (or binary) messages.
In our quote server, we switched from our backend returning a pre-formatted string of a quote, to returning a JSON object so we could get the parts ourselves.
Think about what structure would be useful for our client and our server to know about.
If we’re going to add more messages in the future (e.g. for “liking” a message), how will the receiver of the message know what kind of message the one it receives is?
One thing we often do is wrap our message in an object, with a field specifically saying what the command is.
e.g. instead of sending:
{
"user": "Azin",
"message": "Hello!"
}
we may send:
{
"command": "send-message",
"message": {
"user": "Azin",
"message": "Hello!"
}
}
This means that if we add new commands in the future, we don’t need to change our existing code.
Think about timings
When we first load a page, we need to get all of the messages that already exist.
After that, we can ask to be notified of new messages.
There are a few ways we could do that. An interesting question is what happens between these events?
Imagine we made an HTTP GET request to ask for all of the messages, then created a WebSocket to get new messages. What happens if someone sent a message between when we got our response, and when the WebSocket was connected? How can we make sure we don’t miss any messages?
Or imagine we made a WebSocket request, and expected to receive a list of all previous messages, and then to keep receiving updates. Does the server need to remember which messages have already been sent to each client?
Exercise
Remember WebSockets are bi-directional
Now, we’re using a POST request to send a new message, and a WebSocket to stream receiving new messages. But we know that WebSockets are bi-directional - we can both send and receive information on them. We could change our sending logic to also use our WebSocket. Or we could keep using HTTP POST requests. Both of these approaches work.
Exercise
Think: What advantages does each approach have?
Why might we want to change our implementation to use a WebSocket for sending messages?
Why might we want to keep using POST requests for sending messages?
Why might we want to support both on our server? Why might we only want to support one on our server?
👍👎 Adding like/dislike
Learning Objectives
The last requirement we have for our chat application is the ability to like/dislike a message (and see what messages have been liked/disliked).
Exercise
Think about what information a client would need to provide to a server in order to like/dislike a message.
Think about what information a server would need to provide to a client in order to display how many likes/dislikes a message has.
Think about what information a server would need to provide to a client in order to update how many likes/dislikes a message has.
Write these things down.
Identifiers
One of the key new requirements to add liking/disliking a message is knowing which message is being liked/disliked. When a client wants to like a message, it needs some way of saying this is the message I want to like.
This suggests we need a unique identifier for each message:
- When the server tells a client about a message, it needs to tell it what the identifier is for that message.
- When a client tells the server it wants to like a message, it needs to tell it the identifier for the message it wants to like.
- When the server tells a client a message has been liked, it needs to tell the client which message was liked, and the client needs to know enough about that message to be able to update the UI.
Message formats
Your server is now sending multiple kinds of updates: “Here’s a new message” or “Here’s an update to the number of likes of an existing message”. You will need to make sure the client knows the difference between these messages. Your client will need to know how to act when it receives each kind of message.
Changes or absolutes?
When new likes happen, the server needs to tell the client about it. We need to choose how the server will tell the client about this. Two options:
- The server could tell the client “this message was liked again”.
- The server could tell the client “this message now has 10 likes”.
Both of these can work.
Exercise
Write down some advantages and disadvantages of a server -> client update being “+1 compared to before” or “now =10”.
Choose which approach you want to take.
Exercise
Implement liking and disliking messages:
- If a message has a non-zero number of likes or dislikes, the frontend needs to show this.
- The frontend needs to expose some way for a user to like or dislike any message.
- When a user likes or dislikes a message, the frontend needs to tell the backend about this, and the backend needs to notify all clients of this.
- When a frontend is notified by a backend about a new like or dislike, it needs to update the UI to show this.
You may do this in your polling implementation, WebSockets implementation, or both.
🔎 Identifying common functionality
Learning Objectives
Often in different HTTP handlers we do the same thing. This happens because we use similar patterns. For instance, we may expect all of our request bodies to contain JSON objects. Or we may expect all requests from signed in users to have an Authorization header in a certain format.
For instance, an implementation of “receiving messages” and “receiving reactions to messages” may look like this:
| |
(It’s ok if your implementation looked different, but we’re going to talk about this example).
Both of these POST handlers mostly contain code for parsing a response body as JSON.
This causes a few problems:
- Imagine we found a bug in our parsing - we’d need to fix that bug in multiple places.
- It’s really hard to tell, at a glance, what each handler is actually doing.
Let’s try to work out what is specific to each handler, and what is more general shared behaviour.
Specific vs shared behaviours
Both handlers accumulate chunks of the request as a string, then parse them as a JSON object, returning an error if this failed:
| |
Both implementations check that the object contains certain keys, but they each check for different specific keys. These are doing the same thing, but with different parameters:
| |
Then each implementation does something quite different. The highlighted code is very specific to what the endpoint is for:
| |
Ideally, most of the code in these functions would be specific to the behaviour of the handler. It would be easy to see each function does from reading them.
✂️ Extracting common functionality
Learning Objectives
Having identified these three parts of our handlers, we can extract functions to make them more clear.
One way we could do this is:
| |
Why is this better?
Before, the /message function body was 20 lines. 18 were just for processing the post body, and 2 were for the actual functionality. 10% of the function was for relevant functionality.
Now, the function body is 7 lines. They say: “I expect the request to be a JSON object”, “I expect these fields in the object”, and then do the relevant functionality.
We can see at a glance what the function does. And the common functionality has a clear name explaining what it’s doing, rather than needing us to read 18 lines to work it out.
We can also more easily see the difference between what the two endpoints do. Each endpoint expects the request to be a JSON object. They both expect some fields, but different fields. Then they do something different with them. This is much easier to see than before, where a subtle difference may have been hard to notice.
We can also ignore the details of the helper functions if we don’t need to know them. If we don’t care exactly how the body is parsed as JSON, we can just trust that the function does it somehow. If we do need to know, we can look into the function. If we need to change or fix something, we only need to do it in one place.
➡️ Continuation styles
Learning Objectives
We extracted functions from our code. We saw three different ways of deciding what to do next:
app.post("/message", (req, res) => {
parseRequestAsJsonObject(req, res, (req, res, body) => {
if (!ensureFields(body, res, ["user", "message"])) {
return;
}
receiveMessage(body.user, body.message);
res.send("ok");
});
});
app.posttakes a callback which expects two arguments: a request, and a response. It resolves to a completed state whenres.sendis called (which may be done asynchronously).parseRequestAsJsonObjecttakes a callback very similar toapp.post’s, but which also takes abodyparameter.ensureFieldsitself callsres.sendif there was an error, and returns a boolean indicating whether the function should keep going or stop.
Each of these approaches is doing something similar, but they’re all a little different.
Let’s look at a few different ways we could’ve written ensureFields and parseRequestAsJsonObject:
More callbacks
ensureFields could’ve taken a callback to call if the fields were correct:
app.post("/message", (req, res) => {
parseRequestAsJsonObject(req, res, (req, res, body) => {
ensureFields(req, res, body, ["user", "message"], (req, res, body) => {
receiveMessage(body.user, body.message);
res.send("ok");
});
});
});
This works, but lots of callbacks can get quite hard to read and follow.
We’ve seen a similar problem before with asynchronous code. And we’ve seen Promises, and async/await as ways of solving that problem.
Unfortunately, the reason we need callbacks here is different, and so can’t be solved exactly the same way. The problem here is that a function cannot tell the function that called it to stop running. That’s what we’re doing in the original code with:
if (!ensureFields(body, res, ["user", "message"])) {
return;
}
ensureFields really wants to tell the calling function “If the fields aren’t correct, I’ve already rejected the request, you should stop running”. But it can’t automatically do this. So it returns a boolean, and we need to check it and decide to call return ourselves.
We could use Promises to manage our callbacks a bit better, but we can’t use async/await because it doesn’t have a way to say “return early and stop running”.
Hiding information in existing parameters
parseRequestAsJsonObject takes a callback which takes three parameter: req, res, and body.
Alternatively, we could’ve written parseRequestAsJsonObject in such a way that it would add information to the req parameter, rather than require an extra one:
app.post("/message", (req, res) => {
parseRequestAsJsonObject(req, res, (req, res) => {
ensureFields(req, res, ["user", "message"], (req, res) => {
receiveMessage(req.body.user, req.body.message);
res.send("ok");
});
});
});
Here parseRequestAsJsonObject would add a property named body to the req object it was passed, which callbacks can access as req.body.
In some ways, it’s really useful to have this third parameter. It means it’s obvious to us which functions expect to be able to access the parsed body. It also makes it obvious that we can’t use our callback without going via parseRequestAsJsonObject - i.e. we can’t write:
app.post("/message", (req, res) => {
receiveMessage(body.user, body.message);
res.send("ok");
});
Because there is no variable called body - we can only use this in a callback which declares body.
In other ways, it’s annoying that the
app.post("/message", (req, res) => {
parseRequestAsJsonObject(req, res, (req, res, body) => {
checkAuthentication(req, res, body, (req, res, body, username) => {
// Handle the request here
});
});
});
Or would we write:
app.post("/message", (req, res) => {
checkAuthentication(req, res, body, (req, res, username) => {
parseRequestAsJsonObject(req, res, (req, res, username, body) => {
// Handle the request here
});
});
});
Because we change the signatures here, the order actually matters. These two functions which extract independent information need to be aware of each other and can only be called in a particular order. Maybe in some of our endpoints we only want to call one of them, but they’re now tied together. These
But using the results of those may not be orthogonal, e.g. if we’re editing a post, and the request body contains the ID of the post we’re trying to edit, the user you’re authenticated as may affect whether you’re allowed to edit that message.
Modifying the existing parameters (e.g. setting req.body) can keep our code more orthogonal. But it also makes some behaviour implicit rather than explicit: taking an argument as a parameter makes it explicit that you expect that parameter to be passed. Just calling req.body and hoping body has been set hides the requirement that parseRequestAsJsonObject was already called. Including requirements in function signatures makes explicit the needs of the function.
Control over next steps
Let’s look at three different ways ensureFields could be written:
Complete control over next steps
ensureFields could take a callback to call on success, and could reject the request on failure. It would have complete control over what happens next for this request:
const ensureFields = (req, res, expectedFields, callback) => {
const missingFields = [];
for (const expectedField of expectedFields) {
if (!(expectedField in req.body)) {
missingFields.push(expectedField);
}
}
if (missingFields.length > 0) {
const joinedExpectedFields = expectedFields.join(", ");
const joinedMissingFields = missingFields.join(", ");
console.error(`Failed to extract fields ${joinedMissingFields} from post body: ${bodyString}`);
res.status(400).send(`Expected body to be a JSON object containing keys ${joinedExpectedFields} but was missing ${joinedMissingFields}.`);
} else {
callback(req, res);
}
};
Responsible for rejecting but not continuing
const ensureFields = (req, res, expectedFields) => {
const missingFields = [];
for (const expectedField of expectedFields) {
if (!(expectedField in req.body)) {
missingFields.push(expectedField);
}
}
if (missingFields.length > 0) {
const joinedExpectedFields = expectedFields.join(", ");
const joinedMissingFields = missingFields.join(", ");
console.error(`Failed to extract fields ${joinedMissingFields} from post body: ${bodyString}`);
res.status(400).send(`Expected body to be a JSON object containing keys ${joinedExpectedFields} but was missing ${joinedMissingFields}.`);
return false;
} else {
return true;
}
};
Responsible for deciding, but not rejecting or continuing
const ensureFields = (req, res, expectedFields) => {
const missingFields = [];
for (const expectedField of expectedFields) {
if (!(expectedField in req.body)) {
missingFields.push(expectedField);
}
}
return missingFields;
};
Exercise
Compare these three approaches. What advantages does each have? What problems do they have? When would each be better to choose?
Write down your thoughts.
In class this sprint you will discuss this.
⛓️ Extracting a middleware
Learning Objectives
It is very common that in an application there are operations we want to perform for most or all requests.
For instance, we often want to check authentication details for every request to an endpoint which requires authentication.
And typically we use the same format for all POST request bodies to an application (often JSON, but some applications use other formats) - we want to parse all POST request bodies in that format.
It can be annoying to have to write the same code (like calling parseRequestAsJsonObject) in every handler. It can also be dangerous to require doing so:
⚠️Warning
If we forget to call a function to check a user is logged in in one endpoint, that may be a big security problem.
One strategy to improve this is to use a
Reading
Exercise
Write a tiny Express application. You must write two separate middlewares.
Requirements:
- There must be an endpoint which handles POST requests.
- A middleware should look for a header with name
X-Username. If this is set, it will modifyreqto add ausernameproperty set to this value. If it is not set, the property should be set tonull. - A middleware should parse the request POST body as a JSON array. It should modify
reqto add abodyproperty to this value. If the POST body was not a JSON array, or the array contains non-string elements, it should reject the request. - The response should look like:
You are authenticated as Ahmed.
You have requested information about 4 subjects: Birds, Bats, Lizards, Bees.
or
You are not authenticated.
You have requested information about 1 subject: Bees.
or
You are authenticated as Gemma.
You have requested information about 0 subjects.
You can test your application by running some curl commands like:
% curl -X POST --data '["Bees"]' -H "X-Username: Ahmed" http://localhost:3000
You are authenticated as Ahmed.
You have requested information about 1 subject: Bees.
⛓️ Using existing middleware
Learning Objectives
We don’t always need to write our own middlewares. For common tasks, such as parsing JSON request bodies, there are existing middlewares we can re-use.
Exercise
Make a copy of your previous middleware application.
Delete the middleware you wrote that handles the JSON request POST body.
Switch to instead the JSON middleware built in to Express.
Debug why the curl command suggested in the previous exercise doesn’t work. Fix this problem by modifying the curl command.