News Posts

Posted on Sun 2018.07.01

Feeds down for a little while.

I'm migrating database backing volumes. The feed feeder database is down at the moment, and it'll probably take several more hours for the copy to complete.

Posted on Mon 2017.10.16

Mitigating Qidian's annoying feeds

If you follow any of the novels posted by Qidian, you've probably noticed the annoying interstitial pages they publish for their RSS feeds, as well as the fact that the latest few chapters of every series are hidden behind a watch-an-ad/paywall.

This is annoying, as it generally means you're a few chapters behind what the site thinks is the "latest" chapter, because you can't actually view those chapters yet, making one's watches somewhat confusing.

As such, I've implemented a feed parser module that both unwraps the feed-interstitial page, and properly ignores releases you can't actually view. This should make following Qidian series much more pleasant, because you won't be falsely alerted for new chapters that aren't yet actually available.

Posted on Fri 2017.06.30

Feeds will be down for a little while (possibly up to a day or so). Ok, normal service is resuming.

I'm doing more reconfiguration of hardware, so the feed scraper system will be inactive while I physically move hardware in my server closet around for a little while.

Sorry about that!

Server restructuring is mostly complete. I have a few things I still want to change, but they're going to have to wait because I don't have all the parts I need on-hand.

Posted on Fri 2017.04.14

Feeds are down due to a server failure Normal Service should resume.

The message broker server that backs the feed system is currently non-responsive, and I'm waiting on the server host to perform an actual, physical intervention.

I'm not sure when things will be fixed, but things may be down for a day or two.

Ok, normal feeds should be resuming now. The issue should be resolved.

For the technically inclined, the problems resulted from the combination of a bug in RabbitMQ (the message brokeer the feeds use), and an issue with the linux tool mountall. The RabbitMQ issue (specifically the fact that the "Shovel" plugin caused it to get stuck on exiting) caused me to simply decide to reboot the server. The issue with mountall then caused the server to not start up properly after reboot (it was stuck waiting for a partition that doesn't exist to mount).

I've removed the shovel plugin entirely, and added a work-around to fstab to prevent the reboot blocker issue, and it shouldn't reoccur.

Posted on Wed 2017.02.15

Feed system will be down or spotty for a day or two for system configuration.

I need to do some large schema changes on the backend of the feed feeder system, which requires taking it offline for a bit periodically PostgreSQL churns.

Unfortunately, the size of the tables in the database make this a many hour, or possibly multi-day process, so things may be shut down for a little while.

Ok, database updates are done, normal service should resume.