This post has been sitting 90% complete for two months, and Iāve been promisng Teepee for around three months; I apologise to all those whose hopes and plans I have been delaying. Well, at least itās here now, and itāll be progressing faster again now.
Background
What is currently rust-http was from the beginning an experimental project. I wrote it for a combination of reasons:
I wanted to use an HTTP library with a strong typing discipline, where things like status codes would actually be status codes, not merely numbers and/or strings.
Having primarily used Django in the past, I had formed a vision of the web framework that I wanted. Shortly after writing down a first draft of that vision, I came across Rust again and assessed it more closely; I quickly decided that it was the first really suitable language for my vision.
ā¦ but Rust had no proper HTTP libraryāāonly a couple of toy libraries already just about to become obsolete, for these were the days when the new runtime was being finished up and the old runtime was still the default.
As a proof of concept (especially in the extensive usage of Rustās type system), rust-http has been successful. Itās evenāādespite its limitationsāāa useful library, as is evidenced by (a)Ā people using it, and (b)Ā no one writing any direct competitor since.
Various things have changed since the initial experimentation. Most notably:
I understand the requirements for high-performance HTTP/1.1 and HTTP/2.0 better (I only understood HTTP/1.1 well before, and I only had general notions of what the significant differences were in HTTP/2.0).
I actually know Rust well now (rust-http was the first non-trivial Rust code I wroteāāit shows).
The old runtime is now completely supplanted by the new runtime. (This happened some time ago and is, truth to tell, scarcely of note as concerns rust-http.)
I/O errors are now explicitly handled with things returning IoResult.
A TcpStream can now be cloned, allowing for storing the TcpStream in both the request and the response for a server (important for squeezing optimal performance out of an HTTP/1.1 server) and also the ability to work with the same stream across multiple tasks (which is crucial for HTTP/2.0).
What comes next for rust-http?
Having assessed the situation and the code extant at length, I have come to the following conclusion: rust-http must be completely rewritten.
It is expedient for this to happen immediately rather than later, to minimise technical debt.
The alternative was to keep a fair amount of the code base intact and alter it gradually into the new, deliberately designed form. This was my preferred approach until recently when I took a more careful look at how the existing structures would map onto what I was designing. When I did that I decided that at the very least the transfer modelāācurrently one layer due to what TcpStream used to be and because I was at the time writing the buffered stream support myself and made the (quite natural) error of putting too much HTTP logic into itāāthis transfer model needs to be two distinct layers (the stream buffer, applicable to the entire connection, and the transfer encoding, applicable to only one request) to ensure it is maintainable, plus a third for the data structures (again, that is presently more or less blended in with the rest). I am convinced that replacing this part at least wholesale is the only sensible plan.
Beyond that, there are various other things which I would like to take this opportunity to redesign properly, such as the headers system. In the end, there is very little that I donāt want to do at least something to. (I believe Iām content with the Status struct, but even it will be redone with a macro rather than source code generation, for added coolness.)
So, then, I believe a complete rewrite is in order. Parts will be used as reference material, but no code will simply be copied over. It is safer thus. I will also be imposing the rule that all code that is written must include meaningful doc comments and must be tested. (Some of the key parts of rust-http are not tested; and bugs in an HTTP implementation can be a nightmare to pinpoint and fix.)
I recognise that in this I am vulnerable to second system syndrome. This is part of the reason why I want my plans reviewed by others before building it all.
Itās becoming a broader project
Now that Iāve become thoroughly acquainted with Rust, Iām more sure of my footing: I want to work more with Rust in the future and I believe it will make a very good platform for web-related work; and, whatās more, I intend to be in the forefront of this brave new world where writing fast, safe and correct websites is actually popular.
I have been mulling over what to do with rust-http. Itās got to the stage where I think separating the client and server into separate crates will be a good planāāthey have some shared things (certain traits and utility methods and all the header types), but beyond that they are largely different. I think that splitting it into httpc (client) and httpd (server) crates, with a common crate containing their shared dependencies and data types (perhaps httpcommon), is probably the way forward (this is one of the points Iām least confident about).
In the end, this project will end up quite a bit like Spray is for Scala, and I think that having an overarching project which manages it all is probably a good idea.
My first desire was http.rs with its eponymous domain name, but alas! RNIDS (the operator of the .rs registry) has reserved http.rs, presumably to avoid any form of confusion.
The name I ended up settling upon was the only one which I came up with that I really liked: Teepee. Teepees are those American Indian tent thingies (also spelled tipi or tepee) but is absolutely nothing to do with Apache HTTP server (that connection was suggested by a couple of people independently afterwardsāāit was not my intent in any way). The real derivation of the name was the latter two letters of HTTP. The project will be based at teepee.rs.
What are the important new features?
Proper multiple HTTP version support. The library will have good support for all HTTP versions: 0.9 (yes! really!), 1.0, 1.1 and 2.0.
Various ābad thingsā will be able to be dealt with, rather than being dropped or otherwise inaccessibleāāe.g. bad headers,
System boundaries will be more accurately segregated, abstracting away HTTPās transfer details more completely than it is presently done and leaving a more sound API.
The tools for working with HTTP at a lower level will be more solid, and the higher level interface will sit cleanly on top of that lower level interface.
In consequence, extending it will be easier; WebSocket, for example, with its CONNECT upgrading, could only be implemented as a kludge in rust-http; such things will have proper support under Teepee.
Unified, pureāRust compilation model (using custom syntax extensions in place of source code generation).
Partially in consequence of all these things, Teepeeās HTTP libraries will be much easier to maintain. (I have decreed it so, and so it shall be!)
Under the umbrella of the Teepee project will also exist implementations of other related specifications like WebSocket and Server-Sent Events.
The new designs
I am still making some of the designs for Teepee. I will post them here as I complete them (or a little staggered); I will also be posting discussions of some of the matters that I think could do with particular discussion.
At this point, everything can be questioned and is subject to change. Comments and criticisms are welcome as I publish the designs.
If you wish to discuss the design, please join #teepee on irc.mozilla.org.
Are you interested?
In order to succeed, Teepee is going to need collaboration. There are a few classes of people that will be particularly valuable at this stage:
Servo developers: Teepeeās HTTP client needs to meet Servoās requirements.
HTTP implementers: for both the client and the server, to ensure everything (client and server) is implemented in a reasonable manner.
Web app developers: HTTP server designāāis it appropriate for the forms of servers that you want to write? (Iāve also got to be careful about the web framework story, that things are built in a way that doesnāt end up with frameworks reimplementing most things too much, or that it forces too much on you. Thatās part of the reason for having a more layered approach this time round.)
If you have any feedback on rust-http as it is at present and things you like about it, donāt like about it, canāt stand about it, couldnāt bear to see changed about itāāplease speak up.
If youāre interested in getting involved with Teepee or have opinions to express, please join #teepee on irc.mozilla.org or drop me a line.
A little bit of personal plugging
In the long term, I hope the Teepee project as a whole to be both directly and indirectly a source of income for me and others in many and various ways (e.g. consulting, support, sponsorship of features). Whether or not this will pan out remains to be seen. At present Iām operating as a freelancer, spending most of my time working with a certain web agency (working mostly with Django), but I would love to spend more time working in/on Rust, rust-http and Teepee. If you should happen to want any development done in Rust or with rust-http, or should out of some extraordinary and inexplicable generosity desire to sponsor work on Teepee, Iām ready!
Actively soliciting donations in this way is not something that I have observed in the Rust community yet, so Iāll make myself a test case of what the response is and whether anything happens. (Even if itās not quite a representative case, for by my observations good HTTP support is the biggest thing people want in Rust, beyond the language itself.)
If you want to hire or contract me for anything, please email me@chrismorgan.info.
If you want to donate to me directly to work on these things:
Finally: yes, this article is rather light on actual details, especially of the design. These will come in the coming weeks.