There has been an explosion of podcasting stories around, and one of the absolute best that I’ve found so far can be found here. There is lots of good stuff to absorb here, including equipment reviews of radios, tips for producing radio programs, and reviews of public radio programs. I haven’t listened to his podcast yet, but if it is anywhere close to the quality of the rest of his site, we are in for a treat.

Here is a little tip that I just figured out. I was trying to use a smart playlist to create a special playlist which contained all of my most favorite daily podcast shows that have arrived in the last three days. The normal smart playlist interface allows you to specify a list of conditions, but they are all connected with either an AND or an OR. For instance, what I wanted to do is create a playlist for all shows added in the last three days, AND is one of ‘Evil Genius Chronicles’ OR ‘Linux Link Tech Show’ OR ‘IT Conversations’.

The trick is to used nested playlists. Create one smart playlist which contains all the shows that you find interesting. Then create a second playlist which specifies that ALL the conditions must be met, and add a rule which is “Playlist is My-Favorites” and another rule which is “DateAdded is in the last three days”. Voila!

I didn’t invent this idea, but I did find it useful, and hope you do too.

Where your host tries to get in touch with his inner geek, ponders the world of functional programming, and then rambles about VOIP and Asterisk.

• I like Geek News Central. They have begun podcasting too.
• You can learn about Haskell, Ocaml and Standard ML on the web.
• I’m taking advantage of dreamhost.com’s 7th anniversary deal to provide some more server bandwidth for my podcasts. Let me know if you have any difficulties downloading them by sending me email.
• Binary Revolution – The Revolution will be Digitized! — Episode 60 has a nice bit about VOIP and Asterisk.
• The Asterisk Home Page
• A useful VOIP site
• The Darker Side already have a real phone number that you can dial in and leave comments for their show.

While digging around, I found out about mod_torrent, a project which appears to be moribund, but also mod_bt: making things better for seeders, which appears to be alive. Both are Apache modules which are supposed to be able to transparently server files via bittorrent. I think I might have to give this a try. I’ll let you know when/if it works and is ready for testing.

Well, my experiments with podcasting are revealing a couple of problems with the current podcasting solutions, and I thought I’d write a couple of them down so that others can comment and/or integrate them into their future developments.

Please, please, please, set the user agent fields in your ipodder requests…

Programmers out there, please set the user agent field in your http request, including a version number and a web address where we can learn more about your client. I’ve had a number of clients who seem to access my podcast directory in a particularly unfriendly way, often repeatedly downloading the same file up to a dozen times. I would like to be able to determine if I am doing something wrong or potentially report a bug to your client, but I can’t if I cannot identify which client you were using.

Be a polite client

Please, use ETag and Last-Modified tags. Try to do something sensible with all status codes that are returned from the HTTP server. In particular, make sure that you act appropriately with redirects, accept status codes like 503 (resource unavailable) and read the suggested Retry-After response code, and handle ranges and partial downloads. Take some time to really make the HTTP transactions processing in your client bulletproof.

Scheduling ipodder updates…

Most ipodder clients now have way of scheduling the times when an ipodder polls and downloads new content. I’m going to put forth the bold suggestion that most of these are actually bad: the reason being that certain intervals are too short to be friendly to the remote site’s bandwidth, and they tend to cluster around the top of the hour, and cluster around certain common hours. I’m beginning to see this fairly clearly in my access logs. What this means is that occasionally my http server is completely idle, and other times people are contending for my bandwidth in trying to download my latest/greatest podcast.

A much more polite system would be for the downloading script itself to pick random times to update, and at infrequent intervals. This would help even out the access patterns at the server, and would make for better download speeds at the client side. If your client properly handles the status code 503, you can even have your client back off and retry (perhaps with exponentially decaying frequency ala Ethernet).

Addendum: It is probably counterproductive to do this with BitTorrent. In that world, you actually want lots of people to collide at the same time, so that you can take maximum advantage of the parallelism available. Perhaps your webserver should deny requests for BitTorrent feeds until it reaches some threshold, and schedule them (perhaps using RetryAfter) to all come back at some later time to create a more efficient Torrent Network.

Integrating functionality to the http server

Some of the above requires cooperation from the http server, so why not integrate the functionality directly into an http server. If you pick a high performance but simple server like thttpd or boa, you could (at least in theory) directly implement throttling, access control and perhaps even P2P functionality like BitTorrent into a single server which would be simpler to setup and to manage.

Just some ideas and hints. Feel free to comment.

I’m getting hammered pretty hard by requests for my “podcast.wmv” file. Anyone want to create a mirror for it? Send me an email.

I passed another minor milestone: according to my script which peruses my http logfiles for downloads of my podcasts, over 100 unique IP addresses have accessed at least one of my audio podcasts. Kudos to me! Long live the king!

But success comes at a price: I host my webpages on my home machine (ssh, don’t tell anyone) and occasionally in the last week, I’ve noticed that I can no longer get any work done on my home machines because the uplink speed of my cable modem becomes saturated. Even my son has noticed this: I suspect that this was the cause of at least one mysterious “power outtage” on my webserver.

The problem is throttling: uplink for my home network is nominally about 300Kbits/second, or roughly 28Kbytes/second. This means that the average 10Mbyte podcast (mine have run a bit longer than this, but this is a back of the envelope calculation) takes about six minutes to download, running my uplink flat out. This means that about 10 people per hour can get my podcasts, or about 250 per day. Probably once the listeners reach half that amount, it probably means that my network uplinks are saturated over half the time, and it will be impossible for me to use my network for any other task. So I’ll be watching my logs to see how close I am to getting to this point.

But let’s imagine for a second that you don’t reach this point, and stabilize at some smallish number of subscribers, well within the capacity of your network pipes. How can you help reserve some of your total bandwidth to make sure you can get access to your machines without h-a-v-i-n-g t-o w-a-t-c-h c-h-a-r-a-c-t-e-r-s c-o-m-e o-u-t 1 p-e-r s-e-c-o-n-d?

Well, I figured out a way.

On yesterday’s blog, I mentioned that I was trying to figure out a way to throttle the bandwidth used by Apache to send mp3 files so that some guaranteed headroom would be available for things like ssh. Years ago, I ran thttpd, the throttling http server written by Jef Poskanzer. It was cool for many reasons (small size, fast, portable, easy to configure and secure) but had one unique feature: you could specify a file of regular expression patterns and a maximum and minimum bandwidth that you wanted the server to dedicate to files which matched that URL. While thinking aloud on my blog, I wished that there was similar functionality in Apache. I run WordPress as my weblog, which requires PHP and that’s something thttpd can’t handle, at least without experimental patches.

But then I realized there really was a simpler solution: use Apache to serve my weblog, and thttpd to serve mp3 files.

To make the idea more concrete, I store all my brainwagon.org audio files in a particular directory, which can be referenced as https://brainwagon.org/audio, so my enclosure url’s all look like https://brainwagon.org/audio/2004-10-04.mp3. Let’s imagine that I change the URL to https://brainwagon.org:8080/audio/2004-10-04.mp3, and run thttpd to serve the same files. Now, I can create a very simple throttle file:

# Slow down the download of mp3 files to keep from 
# choking our uplink.
#
**.mp3		20000		# 20K per second, even locally.

From the thttpd documentation:

Throttling is implemented by checking each incoming URL filename against all of the patterns in the throttle file. The server accumulates statistics on how much bandwidth each pattern has accounted for recently (via a rolling average). If a URL matches a pattern that has been exceeding its specified limit, then the data returned is actually slowed down, with pauses between each block. If that’s not possible (e.g. for CGI programs) or if the bandwidth has gotten way larger than the limit, then the server returns a special code saying ‘try again later’.

The minimum rates are implemented similarly. If too many people are trying to fetch something at the same time, throttling may slow down each connection so much that it’s not really useable. Furthermore, all those slow connections clog up the server, using up file handles and connection slots. Setting a minimum rate says that past a certain point you should not even bother – the server returns the ‘try again later” code and the connection isn’t even started.

Cool!

Now, the maximum amount of bandwidth used by serving mp3 files will be 20Kbytes/sec, or roughly 70% of my total bandwidth. This should leave enough headroom for my interactive sessions to remain snappy. The headroom means that the download will take about eight minutes instead of only six, but we never max out the channel.

I’ll probably start serving my files this way beginning with my next audio post.

Engadget has a nice article on getting and making podcasts. It’s mostly Mac-centric, but still could be of use to others.