News Posts

Posted on Sun 2018.12.02

More Qidian issues.

So Qidian has apparently decided to shit their own bed, and moved a bunch of content behind a paywall.

This means a lot of the Qidian links are now broken.

I should have a system that will automatically clear out chapters that aren't actually available (or ones that have been retroactively paywalled, which seems to be most everything) up in a day or two.

Posted on Sat 2018.11.17

Large-Scale OEL content cleanup.

I'm doing a big cleanup of some of the more poorly-matched OEL content.

Basically, there was a period of time when the initial OEL feeding system from RoyalRoadL didn't properly handle series with the same name, by different authors (I originally assumed RoyalRoadL wouldn't allow duplicate series names).

Anyways, the end result is there are a bunch of OEL series where a single name is associated with multiple author names, and has all the releases for all the different series sorted together, being no use to anyone.

Anyways, I'm now doing a big batch cleanup.What it will do is basically delete everything but the series-name entry for any OEL series with more then one author. It will also remove every author but one from the relevant series. The current series feeder system should then re-create any associated metadata entries the next time it walks the series, as well as create separate series entries for series for each author (other then the one left on the cleaned-up series entry).

This has the unfortunate side-effect of possibly breaking any reading list entries if people are watching a series that gets cleaned, and the particular variant of the series they are interested in is not the name that is randomly chosen to remain associated with the original series link.

Anyways, it should result in a more correct end-result, with series chapter listings that are actually useful.

Posted on Wed 2018.09.05

Feed Weirdness

Something strange is going on with the RSS feed parsing system. I'm not sure what, but it appears that it is falling behind.

I think it might be a priority inversion in the webspider queue management system, but I haven't nailed it down completely, yet. Anyways, updates may be spotty for a few days until I figure out what caused what.

Posted on Sat 2018.08.11

Normal service is now resuming.

I've resolved the issue with broken updater. I had to re-provision the private SSL certificates used for the RabbitMQ broker communications.

Anyways, the RabbitMQ host server is now on Ubuntu 16.04 LTS, so it shouldn't need much for another few years other then regular updates as well.

Posted on Sat 2018.08.11

Updates are currently broken

Basically, I rebooted the rabbitmq server that manages the update communications from the web-spider backend, and rabbitmq didn't come back. It turns out that there are unhandled version incompatibilities between the latest version of erlang and the latest version of rabbitmq that is packaged by the rabbitmq devs for ubuntu 14.04.

Anyways, I'm in the process of updating the rabbitmq server, but distribution updates are always kind of dangerous, particularly when (as in this case) you use a lot of non-standard apt-get repositories. We'll see what happens.