Scraping comments on HN hits the robots.txt wall rather quickly.
At a maximum 2 scrapes per minute, your cache will be 60 minutes old when you're watching 120 (single-page) comment threads. That might be tolerable. Now try 1200 or 12,000 comment threads.
You can do smart cache refreshes, but you will miss bursty activity. You can ignore articles older than X days, but you'll miss the "dupe, comments here:" resurgences.
HN gets about 150 posts per day that hit the top 60, at least briefly, as a point of reference.
This is a neat thing, but if it gets popular you will have a problem. On the plus side, it's self-limiting. As the usage gets higher, the experience will degrade and usage will decline. No intervention required by you. :)
I've wished for a policy-compliant way to cache comments in a timely manner, but there's just no way to stay current under the existing rules.
You can't scrape with AJAX because of cross domain security restrictions.
One potential solution to obey robots.txt might be to spawn multiple small EC2 instances with different IP's and have them coordinate with each other to share the crawling without individually running over the limits. (This is also useful for scraping from sites that have rate limits)
robots.txt doesn't enforce itself so there is no IP limitation; this is still a violation and no better than simply lowering the delay on a single scraper.
Perhaps an iframe would work best. With some injected styles and javascript (some that would make the iframe expand automatically à la Disqus), I think this would work nicely.
At a maximum 2 scrapes per minute, your cache will be 60 minutes old when you're watching 120 (single-page) comment threads. That might be tolerable. Now try 1200 or 12,000 comment threads.
You can do smart cache refreshes, but you will miss bursty activity. You can ignore articles older than X days, but you'll miss the "dupe, comments here:" resurgences.
HN gets about 150 posts per day that hit the top 60, at least briefly, as a point of reference.
This is a neat thing, but if it gets popular you will have a problem. On the plus side, it's self-limiting. As the usage gets higher, the experience will degrade and usage will decline. No intervention required by you. :)
I've wished for a policy-compliant way to cache comments in a timely manner, but there's just no way to stay current under the existing rules.