Sockets Part II

The previous post should give a good idea about how two programs can chat, but the example was really trivial. In this post, I'd like to progress to make it into something more realistic by taking in some real world requirements.

Servers need to listen constantly
The greeting program as it stands waits for a connection, sends a message to any program that joins and then quits. That's not very useful, so to progress toward something that is more useful we need to be able to accept any number of requests. I can do this easily by adding a loop so the server looks like this:

        static void Main()
        {
            var thisMachine = Dns.GetHostAddresses("")[0];
            var server = new Socket(thisMachine.AddressFamily,                   SocketType.Stream, ProtocolType.Tcp);
            server.Bind(new IPEndPoint(thisMachine, 4510));
            server.Listen(10);

            while (true)
            {
                var client = server.Accept();
                client.Send(Encoding.ASCII.GetBytes("Hello"));
            }
        }

This is how any real server is going to operate, by looping and handling requests from clients until someone kills it.

Chatting is a two way thing
The server is not limited to sending out messages, it's a two way interaction so the client can also send messages. Let's change the server to accept a message from a client and react differently depending on the message.

while (true)
{
    var client = server.Accept();
    var request = new byte[5];
    client.Receive(request);

    switch (Encoding.ASCII.GetString(request))
    {
        case "Hello":
           client.Send(Encoding.ASCII.GetBytes("Hello"));
           break;
        case "Clock":  
           client.Send(Encoding.ASCII.GetBytes(
                      DateTime.Now.ToShortTimeString()));
           break;
    }
}

Must allow parallel conversations
There is a big problem with the above implementation. If I was to create a client that connected and waited a long time before sending a request, or didn't send a request at all, it would block others from connecting. The normal way to get around this is to create a separate thread for each client, but C# also offers tasks so I'm going to use that instead.

I'm going to reduce the loop to this:

            while (true)
            {
                var client = server.Accept();
                Task.Factory.StartNew(() => Chat(client));
            }

and move the chatting to another method to handle each request.

        private static void Chat(Socket client)
        {
            var request = new byte[5];
            client.Receive(request);

            switch (Encoding.ASCII.GetString(request))
            {
                case "Hello":
                   client.Send(Encoding.ASCII.GetBytes("Hello"));
                    break;
                case "Clock":
                   client.Send(Encoding.ASCII.GetBytes(
                       DateTime.Now.ToShortTimeString()));
                   break;
            }
        }

Now each chat runs in its own task and won't block. And that's it, the basic workings of a server.

Sockets Part I

This post gives an overview of how computer programs can chat with each other. I can't claim to have any great knowledge in this area, I'm just interested enough to try to learn by writing about it. So my first warning is that there are probably mistakes. My second is that this will be a basic overview, my assumption is that anyone reading this knows very little of the area.

First off, there are some obvious and important things to point out.

You need to know the location
If I was writing a program and I wanted two separate classes / functions etc. to interact I would give one a reference to another and let it call what ever it wants. It's pretty simple. With two separate programs it's not that simple because there is no way of knowing where in memory the other program is, and in reality it's probably in the memory of another computer running in some other location in the world. That's the first problem, I need to know where the program is so I can chat with it. This seems to basic to point out but it's good to make things clear.

The solution to this is the IP address and port number. I can use the IP address to identify any computer in the world. Once I know, there are thousands of ports on that computer with each one capable of having a program listening on it. So if I want to chat with another program, I need to identify the machine it is on with the IP address and what port number its listening at.

You need to agree how to transport the data
The messages sent between the programs need a transport mechanism, some low level piece of software that moves it around. For the example below I'm using TCP as the transport layer. TCP guarantees me that a message will be received at the cost of performance. Another option is UDP, which does not guarantee delivery but benefits from being quite quick because of this. The general rule I've come across here is use UDP when you want performance and TCP when you want reliability. That's why VOIP tends to use UDP and web browsers tend to use TCP.

You need to be talking the same language
Chatting within a single program gives you all of the advantages of types. One things gives a number to another, you can be guaranteed that the other will receive a number and can do all sorts of numerical things with it. When chatting between two programs, there are no numbers or strings, only streams of bytes. Each byte represents a character encoding from a character set and both programs need to agree on the character set that they will use.

There's lots to learn in this area (see here http://www.joelonsoftware.com/articles/Unicode.html or here http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF) but the important thing is that both programs are using the same character set.

Working through an example
To keep this simple I want to show as little as possible in each step. First is to show that the two programs can chat. I'll create a C# server that will listen for a connection, greet a client that connects by saying 'Hello' and shuts down. Sounds trivial, but there's quite a bit going on.

static void Main()
{
  var thisMachine = Dns.GetHostAddresses("")[0];
  var server = new Socket(thisMachine.AddressFamily,       SocketType.Stream, ProtocolType.Tcp);
    server.Bind(new IPEndPoint(thisMachine, 4510));
  server.Listen(10);

  var client = server.Accept();
  client.Send(Encoding.ASCII.GetBytes("Hello"));
}

And a python client that connects and displays the received message.

        s = socket.create_connection(('', 4510))
        print s.recv(5)
        s.close()

Let's look at what happens in more detail

            var ipAddr = Dns.GetHostAddresses("")[0];

This will get the localhost addresses ("" tells is to use local host). The addresses returned here will be all of the available addresses for this machine, both IPV4 and IPV6, and I'm just picking the first one off the list.

            var server = new Socket(ipAddr.AddressFamily, SocketType.Stream, ProtocolType.Tcp);

Next step is to create a socket. This bit is important because there are two types of sockets, server sockets and client sockets. The socket we are creating on this line is a server socket and it will be responsible for listening and hooking other programs up with client sockets. This socket is setup to chat over TCP using streams.

            server.Bind(new IPEndPoint(ipAddr, 4510));

Telling the server socket where to listen...

            server.Listen(10);

...and to listen for anyone that wants to chat. The number 10 here is the number of clients that we will allow to back up before going kaput. It's very unlikely that we will reach 10 because the server socket does nothing other than listen and...

            var client = server.Accept();

...accept incoming requests. The call to accept here is blocking, so at this stage the server will wait until another program connects to the end point that it is listening to. This will be done by the python client.

            s = socket.create_connection(('', 4510))

Once our python client connects to the correct location, our server can move on.

                Side Note: It's probably obvious that if you use an incorrect IP address or port number that you won't be able to locate the server. What may not be that obvious is that looking for an IPV4 address when the server is at an IP6 address will mean you're looking at the wrong place. When I get the server local address above I pick the first one off the list so I don't know whether it's an IPV4 or IPV6 address. I get around this by using create_connection() rather than create() from the socket in the python program. create_connection() tells it to look at all available address on the machine.

            client.Send(Encoding.ASCII.GetBytes("Hello"));

The server has created a client to talk to the python client from Accept() and uses that client to send the greeting in bytes, because that's the highest form of communication between two programs. Note that the encoding is ASCII, I'm using ASCII because that's the default in python. So we send the string "Hello", encoded in ASCII, as an array of bytes...

                Side Note: Picking the wrong character set can mess things up. If I switch the line that sends the message to be...

              client.Send(Encoding.Unicode.GetBytes("Hello"));

                ... the resulting string in python would be something like 'H<Null>e<Null>l'. This is because Unicode (this is badly named in C#, it should be UTF-16) uses two bytes per character and ASCII only uses one byte per character.

            print s.recv(5)

...and our python client receives 5 bytes and prints the result. And that's it, two programs have chatted.

Fake it till you make it

tldr; fake it till you make it is an underused approach to TDD, but when you can get it to work it gives you the best experience. See Kent Becks book TDD:By Example for a more thorough explanation (this book is highly recommended if you want to understand TDD).

Green bar patterns

In Kent Becks book, TDD: By Example he outlines three approaches to get the tests passing. There's Obvious Implementation, where you just write to code to pass the test. This method works well, but where the implementation is anything more than trivial, it can be difficult to write the correct code. Another is Triangulation, where you create multiple tests to force yourself to write to the implementation. Triangulation is the least recommended by Kent Beck of all patterns because you end up with a lot of redundant tests. Unfortunately, it is probably the most used.

There's a third approach though which I find gives me the best results (when I can get it right) and that's called Fake It (Till You Make It). The idea with Fake It is to hard code the expected result to get to green straight away. Then remove duplication between the test and the implementation.

Using the word wrap example.

I'm going to work through this with the word wrap kata, which is:

Break a piece of text into lines, but with line breaks inserted at just the right places to make sure that no line is longer than the column number. You try to break lines at word boundaries.

First test, should not wrap text that fits.

def test_should_not_wrap_text_that_fits(self):
    self.assertEqual(wrap("text that fits", 20), "text that fits")

and the faked result:

def wrap(text, column):
    return "text that fits"

The test passes. Next step is to remove any duplication between the test and the implementation. Returning the result that the test is expecting is duplication and that can be removed by returning the passed in text:

def wrap(text, column):
    return text

Test passes and there's no more duplication between the test and the implementation.

Doing this wouldn't make a whole lot of sense in the real world. Passing this test is so simple that it would be best to use Obvious Implementation, but I think it serves as a nice introduction to the technique.

Second Test, wrapping text that does not fit on a line.

def test_wrapping_text_that_does_not_fit_on_a_line(self):
    self.assertEqual(wrap("text that does not fit", 15), "text that does\n not fit")

Faking It

It is possible to pass both tests by branching and faking the new result.

def wrap(text, column):
    if len(text) > column:
        return "text that does\n not fit"
    return text

Running the tests should give two passing tests. Now to start removing the duplication. I'm going to do it as slowly as possible to show how the technique can work.

Side note, speeding up and slowing down

The steps in Fake It may seem painfully slow for some, but this is a good thing.  Fake it allows me to do go slowly or quickly, depending on my confidence. At it's slowest I'm making the simplest possible change and running the tests to ensure that I'm at green. I can speed up by taking a few of those steps together. Full speed, where I just make the change I want, is Obvious Implementation. That's the big advantage that this approach has over Obvious Implementation and Triangulation, both of which force me to write all the code at once.

Removing duplication

That returned text is made up of three separate parts, two lines and a line break. After each step here the tests are run to ensure they are still green:

def wrap(text, column):
    if len(text) > column:
        return return "text that will " + "\n" + "not fit"
    return text

That first part is really just as much as will fit in the first column.

def wrap(text, column):
    if len(text) > column:
        return text[:column] + "\n" + "not fit"
    return text

And the second part is the remainder.

def wrap(text, column):
    if len(text) > column:
        return text[:column] + "\n" + text[column:]
    return text

Most of the duplication between the test and the implementation has been removed, but there is still some. That is, it will only handle one line break. It will be easier to remove this duplication if the duplicate return statements are refactored into one.

The first step is the make the return statements the same.

def wrap(text, column):
    wrapped = ''
    if len(text) > column:
        wrapped = text[:column] + "\n"
        text = text[column:]
        return wrapped + text
    return wrapped + text

Once they're the same, the redundant return can be removed.

def wrap(text, column):
    wrapped = ''
    if len(text) > column:
        wrapped = text[:column] + "\n"
        text = text[column:]
    return wrapped + text

By doing this I've changed the structure of the code without changing the behaviour. The solution still needs to be generalised to handle any amount of lines. That's got me thinking that my current test is not good enough. I want to take the approach of changing that if to a while, but I'd like to have some safety in the move.

Triangulation can be useful

This is a situation where triangulation can be useful. I'm going to introduce one more test to ensure that all is good. The test will be wrapping text over multiple lines.

def wrapping_text_over_multiple_lines(self):
    self.assertEqual(wrap("wrapping text over multiple lines", 9), "wrapping \ntext over\n multiple\n lines")

I can pass all three tests now by changing the if to a while and appending to the wrapped text.

def wrap(text, column):
    wrapped = ''
    while len(text) > column:
        wrapped += text[:column] + "\n"
        text = text[column:]
    return wrapped + text

Not done yet

There's one final step I have to take if I want to tidy everything up. When I added that extra test at the end, I also added duplication between the tests because wrapping_text_over_multiple_lines tests the same behaviour as wrapping_text_that_does_not_fit_on_a_line. I can prove this by changing anything in the implementation to break wrapping_text_that_does_not_fit_on_a_line. This will in turn break wrapping_text_over_multiple_lines every time. The same is not true in reverse, so wrapping_text_over_multiple_lines is the test that I want to keep.

Disclaimer

This is not a perfect implementation of word wrap, the intention is not create create a perfect word wrap, just to show how fake it works.
Fake it works really well when you can get it to work, but the biggest difficulty I find is seeing the simplest change to make and sometimes even how or what to fake it. When I'm stuck I end up taking big steps or triangulating to help me move. Often, doing either of these steps will help me understand what I should have done, so I can quickly revert and go back to faking it.

TDD: What should I test next?

tldr; focusing on the problem domain rather than the implementation will help you out

One of the guiding ideas in TDD is to implement the simplest thing that will work. This is good advice, and there is no shortage of posts explaining what it means and why you should do it. But, there's a similar question that occurs before this that always stumps me and that is 'What should I test next?'. The easy answer is the simplest behaviour that the software should have. Always testing the simplest behaviour will allow you to incrementally drive the design. Not testing the simplest behaviour tends to require big leaps in the implementation and makes it easy to get stuck a few tests in.

Initially my thought on this was that the simplest thing was the most basic arguments that I could pass in. It sounds plausible and it also got me moving quickly. The problem was that I normally got stuck after a few tests because I wasn't really analysing the problem.

It's easiest to show this with an example. I'm going to use a TDD kata called word wrap. This is the requirement:

    Break a piece of text into lines, but with line breaks inserted at just the right places to make sure that no line is longer than the column number. You try to break lines at word boundaries.

Taking the approach of using the simplest arguments would go something like this. To keep this short I'll just focus on the string argument.

First test: an empty string, column 10
Second test: a single character, column 10
third test: two characters, column 10

...and on and on. There's probably a lot of different paths you could go with this type of testing but really I was being led by the arguments rather than by the requirement. Taking it back to analysis helped me out. Initially this can be a little harder, but, you're less likely stuck because the problem domain is more likely to lead you in the right direction than the arguments for a method.

So what's, the simplest behaviour that word wrap can have? I can also ask this question in another way, what's the least amount of behaviour that I can add, while still doing something useful? The usual answer to this question for the first test is the 'do nothing' behaviour. In the case of word wrap I would think that this is a when it doesn't need to wrap at all. My specification and test name for this will be 'should not wrap a line that fits'.

The implementation that satisfies this test would look something like this:

public string Wrap(string text, int column)
{
    return text;
}

The difference between this and the previous approach is that I have one test for all lines that fit. Creating multiple tests for lines that would fit would have been pointless because I would have been re-testing the same behaviour. This is also an example of the kind of feedback that you get from your code. Not having to implement any new behaviour for a test is the code saying that you don't need that test.

Disclaimer
In this post I'm advocating the minimum most suitable tests to drive out the design. Tests should also build your confidence though, so if your instinct is telling you to add the extra tests, then go ahead and add them.

What drives the tests in TDD?

tldr; analysis is an under emphasised part of TDD. I'm trying to make a case for developers to pay more attention to this part

Red-Green-Refactor

The introductory mantra in test drive development (TDD) is red-green-refactor. That is, you write a test that fails (red), make the test pass (green) and then tidy up the code (refactor). Following this pushes you toward writing small pieces of behaviour that will incrementally build up to a complete system. Another great advantage of this process is that each test gives you a small problem to solve and that drives the code that you write, hence the name. This sounds great, but trying to put it into practice was very difficult. Initially I'd get stuck trying to think of a test out of thin air, without something to drive it. When I got tired of being stuck and felt that I should start writing something I would think of the tests in terms of a method that a class will have, and then work my way through the possible inputs starting from the most basic. This would get me started but I found that the process wasn't as smooth as I would have hoped and usually got stuck after the second or third test.

Changing the metaphor

Trying to figure out what to test at the level of the test was leading me to the same problems I had when using the code-first technique. I was missing something to drive the test. Then I read Dan North's post on behaviour driven development (BDD)  and things started to make a more sense. He didn't come up with anything new, he just switched the metaphor from test to behaviour. This allowed me to re-frame the problem in terms of something more tangible; I wasn't stuck thinking about tests, I was thinking about the behaviour that the system should have. That still didn't quite do it for me though, because that behaviour has to come from somewhere.

Analysis is important

The driving force for this behaviour is whatever real-world need the software should fulfil. This behaviour still doesn't jump out of thin air though, it requires analysis. This does get pointed out in the BDD posting but it's importance gets overshadowed by the description of the framework. For me the real key is that you perform some analysis and come up with a statement that specifies how the system should work.
The statement plays a very important role because it's the bridge between the real world and the code. The statement forms this bridge by becoming the test name. That's what a test name really is, a specification for a piece of the system, and a series of well written tests will read like a specification document. This is true whether you are doing acceptance tests or unit tests.

Making statements

Each statement should be some true fact about the system. This part sounds easy but becomes difficult when it's tied in with the code in the form of a test name. There are recommended structures for test names, such as Given-When-Then or method-input-result. These can help you out when you are stuck, and in some cases still read OK, but in reality you're writing a specification and your test names help you do this when they in a natural language.

The resulting process looks like this:

Analyse real world problem -> make a statement about the behaviour -> write a test that proves the behaviour -> write an implementation that satisfies the behaviour

Each step gives feedback on the quality of the previous steps. If you find it difficult to make a statement about the behaviour, then you probably need more analysis. If you find it difficult to write a test for the statement, then maybe the statement is too general, or saying too much. If you find it difficult to write the implementation, then maybe there is another statement that you should have made first. There's lots more feedback that can occur between the steps but the important thing is that it is there and will push you back to the analysis phase if necessary.

Disclaimer

TDD is much harder than most books or posts would have you believe. It's hard because it is an analysis and design activity. Rather than giving an easy path to great software, it highlights the complexity by asking hard questions. This is a good thing because you're going to have to tackle this complexity anyway.
I find it difficult to stick to the process all of the time and sometimes I have to go with what I have, whether that's badly named tests or no tests at all. It is worth practising though, because when it works the benefits are huge.

A real life project: The end result

Software projects never really finish, I find I just get to a stage where it's good enough and doing anymore wouldn't add much value so it's time to put this one to bed.

The complete project is available here https://github.com/JohnBrod/PokerBots. It had estimated that I would be finished early this year and unusually that turned out to be accurate.

A little bit on the design

The real challenge in this project was coming up with a design that made sense for the communication between the object I had originally called the dealer and the other programs that were acting as players. The communication went like this:

Dealer tells player to make a bet.
Dealer waits for player to respond.
Player responds and dealer reacts.

This part of the design is set in stone, there isn't any way that I can change this interaction because I'm dealing with the real world. The difficulty was really in the dealer, as well as being restricted to this type of communication the dealer was doing way too much. So, the majority of the thinking involved how to simplify the dealer. The best solution for me was to separate out all these responsibilities into other objects by delegating at every opportunity. This resulted in the dealer being replaced by a number of objects that fulfilled the dealers roles.

For example:

The main entry point is an object that handles playing a tournament. This object will announce the players before delegating to an object that can deal a hand. When the hand is finished the tournament will decide whether to play another hand or not.

This pattern is repeated throughout, all the way to an object that takes bets from the external players.

The resulting communicaton

Communication in the game

PlaysTournament, PlaysHand and TakesBets all wait for responses before continuing. A response comes when the object they have delegated to raises an event to say that it has complete. When they get a response they have a condition that decides whether to initiate another round or finish up.

DealsTexasHoldEm and DistributesWinnings fire and a message and leave it at that, no response is necessary.

There's a lot more going out here, such as how the winnings are decided and how hands are ranked, but for those it's best to read the code.