| *** seongsoocho_ is now known as seongsoocho | 05:23 | |
| * corvus yawns and stretches | 15:49 | |
| corvus | i'll make sure the latest images are pulled | 15:50 |
|---|---|---|
| corvus | and i'll restart zuul now | 15:51 |
| corvus | https://zuul.opendev.org/components is looking good | 15:53 |
| corvus | this startup may take a bit longer because we're populating the connection branch cache in zk one project at a time, and that's taking about 200ms per project. | 15:55 |
| corvus | that should only be on initial startup though | 15:55 |
| corvus | up; re-enqueing | 16:04 |
| corvus | done | 16:06 |
| corvus | debug log chatter looks normal | 16:06 |
| fungi | ooh! | 16:07 |
| corvus | starting zuul01 | 16:07 |
| corvus | expect status page weirdness while it starts | 16:07 |
| fungi | zuul01.opendev.org RUNNING 4.10.5.dev191 e2d6992a | 16:08 |
| corvus | i think it's fully started up now | 16:14 |
| corvus | okay, now we wait :) | 16:15 |
| fungi | there are a few cases in the debug log of "AttributeError: 'NoneType' object has no attribute 'cache_key'" but they're all from well before the restart | 16:15 |
| corvus | oh, i didn't think to clear out the cache before starting. hopefully everything was in a reasonable state. but if we do see an error, one thing we'll want to consider is whether it was due to previously existing bad data. | 16:16 |
| fungi | also "TypeError: 'NoneType' object is not subscriptable" but only prior to the restart | 16:16 |
| fungi | (so far) | 16:16 |
| fungi | the only exceptions i see after the restart are "AttributeError: 'NoneType' object has no attribute 'layout'" but those were occuring before the restart as well | 16:17 |
| corvus | yeah, that's a status_json formatting job running before being loaded | 16:17 |
| corvus | so not something to worry about | 16:17 |
| corvus | i'd just like to share how cool it is to watch both debug logs streaming and seeing the schedulers cooperate :) | 16:18 |
| fungi | some builds for reenqueued queue items have already returned results | 16:19 |
| fungi | (and succeeded) | 16:19 |
| corvus | (it's also really weird to see an idle scheduler but that's the weekend for ya) | 16:19 |
| fungi | yeah | 16:21 |
| fungi | corvus: since the restart, zuul02 has logged two instances of "AttributeError: 'NoneType' object has no attribute 'getRelatedChanges'" but it's in general cleanup so i guess that could involve lingering cache content from before the restart? | 19:12 |
| corvus | thx, i'll take a look after i finish stuffing my face | 19:28 |
| corvus | i'm going to fire up the repl and inspect that error some more | 19:59 |
| corvus | yeah, the issue is that there was an item in the change cache with no data, and it was in the cache before the restart | 20:16 |
| corvus | so i should have cleared the zk state before restarting into this. but also, it might be nice if we could recover from this | 20:17 |
| fungi | makes sense | 20:34 |
| corvus | i think the best thing to do is to clear the zk state and restart; i'm not sure how productive debugging change cache issues across this version boundary is going to be | 20:37 |
| corvus | i'm going to go ahead and restart | 20:37 |
| corvus | stopped; deleting zk state now | 20:39 |
| corvus | starting | 20:46 |
| fungi | sure, thanks! | 20:46 |
| corvus | re-enqueing | 21:08 |
| corvus | and starting zuul01 | 21:08 |
| corvus | all up again | 21:16 |
| corvus | as a data point, i just ran the relevant change key collection routine (which is what was throwing the exception earlier) and it returned without error | 21:17 |
| fungi | oh, awesome, so definitely was all due to preexisting cache content | 21:19 |
| fungi | corvus: there's been at least one more "AttributeError: 'NoneType' object has no attribute 'getRelatedChanges'" since the latest restart | 22:54 |
| fungi | 22:47:59,755 utc in general cleanup again | 22:55 |
| fungi | the only new one so far | 22:57 |
| fungi | identical traceback though | 22:58 |
| corvus | thanks, hopefully we can get better data for this one. | 23:09 |
| corvus | oh interesting; it's the same change as before | 23:19 |
| corvus | i think i see the issue | 23:31 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!