Introducing Teepee: the next step for rust-http

rust-http was but an experiment, an essay in the craft. Here, at last, is the real thing: the Teepee project, a properly engineered HTTP toolkit.

This post has been sitting 90% complete for two months, and Iā€™ve been promisng Teepee for around three months; I apologise to all those whose hopes and plans I have been delaying. Well, at least itā€™s here now, and itā€™ll be progressing faster again now.

Background

What is currently rust-http was from the beginning an experimental project. I wrote it for a combination of reasons:

As a proof of concept (especially in the extensive usage of Rustā€™s type system), rust-http has been successful. Itā€™s evenā€”ā€‹despite its limitationsā€”ā€‹a useful library, as is evidenced by (a)Ā people using it, and (b)Ā no one writing any direct competitor since.

Various things have changed since the initial experimentation. Most notably:

What comes next for rust-http?

Having assessed the situation and the code extant at length, I have come to the following conclusion: rust-http must be completely rewritten.

It is expedient for this to happen immediately rather than later, to minimise technical debt.

The alternative was to keep a fair amount of the code base intact and alter it gradually into the new, deliberately designed form. This was my preferred approach until recently when I took a more careful look at how the existing structures would map onto what I was designing. When I did that I decided that at the very least the transfer modelā€”ā€‹currently one layer due to what TcpStream used to be and because I was at the time writing the buffered stream support myself and made the (quite natural) error of putting too much HTTP logic into itā€”ā€‹this transfer model needs to be two distinct layers (the stream buffer, applicable to the entire connection, and the transfer encoding, applicable to only one request) to ensure it is maintainable, plus a third for the data structures (again, that is presently more or less blended in with the rest). I am convinced that replacing this part at least wholesale is the only sensible plan.

Beyond that, there are various other things which I would like to take this opportunity to redesign properly, such as the headers system. In the end, there is very little that I donā€™t want to do at least something to. (I believe Iā€™m content with the Status struct, but even it will be redone with a macro rather than source code generation, for added coolness.)

So, then, I believe a complete rewrite is in order. Parts will be used as reference material, but no code will simply be copied over. It is safer thus. I will also be imposing the rule that all code that is written must include meaningful doc comments and must be tested. (Some of the key parts of rust-http are not tested; and bugs in an HTTP implementation can be a nightmare to pinpoint and fix.)

I recognise that in this I am vulnerable to second system syndrome. This is part of the reason why I want my plans reviewed by others before building it all.

Itā€™s becoming a broader project

Now that Iā€™ve become thoroughly acquainted with Rust, Iā€™m more sure of my footing: I want to work more with Rust in the future and I believe it will make a very good platform for web-related work; and, whatā€™s more, I intend to be in the forefront of this brave new world where writing fast, safe and correct websites is actually popular.

I have been mulling over what to do with rust-http. Itā€™s got to the stage where I think separating the client and server into separate crates will be a good planā€”ā€‹they have some shared things (certain traits and utility methods and all the header types), but beyond that they are largely different. I think that splitting it into httpc (client) and httpd (server) crates, with a common crate containing their shared dependencies and data types (perhaps httpcommon), is probably the way forward (this is one of the points Iā€™m least confident about).

In the end, this project will end up quite a bit like Spray is for Scala, and I think that having an overarching project which manages it all is probably a good idea.

My first desire was http.rs with its eponymous domain name, but alas! RNIDS (the operator of the .rs registry) has reserved http.rs, presumably to avoid any form of confusion.

The name I ended up settling upon was the only one which I came up with that I really liked: Teepee. Teepees are those American Indian tent thingies (also spelled tipi or tepee) but is absolutely nothing to do with Apache HTTP server (that connection was suggested by a couple of people independently afterwardsā€”ā€‹it was not my intent in any way). The real derivation of the name was the latter two letters of HTTP. The project will be based at teepee.rs.

What are the important new features?

The new designs

I am still making some of the designs for Teepee. I will post them here as I complete them (or a little staggered); I will also be posting discussions of some of the matters that I think could do with particular discussion.

At this point, everything can be questioned and is subject to change. Comments and criticisms are welcome as I publish the designs.

If you wish to discuss the design, please join #teepee on irc.mozilla.org.

Are you interested?

In order to succeed, Teepee is going to need collaboration. There are a few classes of people that will be particularly valuable at this stage:

If you have any feedback on rust-http as it is at present and things you like about it, donā€™t like about it, canā€™t stand about it, couldnā€™t bear to see changed about itā€”ā€‹please speak up.

If youā€™re interested in getting involved with Teepee or have opinions to express, please join #teepee on irc.mozilla.org or drop me a line.

A little bit of personal plugging

In the long term, I hope the Teepee project as a whole to be both directly and indirectly a source of income for me and others in many and various ways (e.g. consulting, support, sponsorship of features). Whether or not this will pan out remains to be seen. At present Iā€™m operating as a freelancer, spending most of my time working with a certain web agency (working mostly with Django), but I would love to spend more time working in/on Rust, rust-http and Teepee. If you should happen to want any development done in Rust or with rust-http, or should out of some extraordinary and inexplicable generosity desire to sponsor work on Teepee, Iā€™m ready!

Actively soliciting donations in this way is not something that I have observed in the Rust community yet, so Iā€™ll make myself a test case of what the response is and whether anything happens. (Even if itā€™s not quite a representative case, for by my observations good HTTP support is the biggest thing people want in Rust, beyond the language itself.)

If you want to hire or contract me for anything, please email me@chrismorgan.info.

If you want to donate to me directly to work on these things:

Finally: yes, this article is rather light on actual details, especially of the design. These will come in the coming weeks.