| sarob | oh, the validate patch isnt updated yet, duhhhh | 00:00 |
|---|---|---|
| lifeless | Alex_Gaynor: never? testr doesn't lock out using a direct runner | 00:00 |
| Alex_Gaynor | lifeless: sorry, I meant for the purposes of what gets run on zuul | 00:00 |
| lifeless | ah | 00:10 |
| lifeless | I don't know for that ;) | 00:10 |
| lifeless | Alex_Gaynor: https://mycloud.rackspace.com/a/rbtcollins/tickets#rax%3Aincident%2Cincident/130722-00002-3 filed about glance | 00:11 |
| Alex_Gaynor | lifeless: ok let's see if a) I can figure out my account creds, b) I can access that | 00:12 |
| Alex_Gaynor | Looks like I can't access that ticket | 00:14 |
| *** sarob has quit IRC | 00:18 | |
| *** sarob has joined #openstack-infra | 00:19 | |
| *** sarob has quit IRC | 00:21 | |
| *** sarob has joined #openstack-infra | 00:21 | |
| *** sarob has quit IRC | 00:22 | |
| *** sarob has joined #openstack-infra | 00:23 | |
| *** ewindisch has joined #openstack-infra | 00:24 | |
| *** sarob has quit IRC | 00:27 | |
| *** ewindisch has quit IRC | 00:42 | |
| openstackgerrit | Chuck Short proposed a change to openstack-dev/hacking: python3: Use six.moves for ConfigParser https://review.openstack.org/38092 | 00:56 |
| *** sarob has joined #openstack-infra | 01:00 | |
| *** sarob has quit IRC | 01:04 | |
| *** dkliban_ has joined #openstack-infra | 01:08 | |
| openstackgerrit | Chuck Short proposed a change to openstack-dev/hacking: python3: Fix unicode and gettext. https://review.openstack.org/38093 | 01:13 |
| lifeless | Alex_Gaynor: 'Confirm Solved' | 01:16 |
| lifeless | 'there is no public glance endpoint' | 01:16 |
| lifeless | :( | 01:17 |
| Alex_Gaynor | lifeless: Someone (who admittingly has the crappy job of dealing with support tickets on sunday) has a very interesting definition of "solved" | 01:17 |
| lifeless | Jaime Carmona | 01:17 |
| lifeless | apparently | 01:17 |
| zul | fanatical support ;) | 01:18 |
| fungi | lunatical support | 01:21 |
| lifeless | I suspect that the ticketing system closes anything when answered | 01:21 |
| lifeless | unless explicitly overridden - when I replied it toggled to open | 01:22 |
| *** dguitarbite has joined #openstack-infra | 01:34 | |
| *** CaptTofu has joined #openstack-infra | 01:49 | |
| *** sgviking has quit IRC | 02:00 | |
| *** sgviking has joined #openstack-infra | 02:00 | |
| *** sarob has joined #openstack-infra | 02:01 | |
| *** yaguang has joined #openstack-infra | 02:02 | |
| *** sarob has quit IRC | 02:06 | |
| *** dkliban_ has quit IRC | 02:19 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Replace setuptools_git with a smarter approach https://review.openstack.org/30522 | 02:30 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: We force installs via pip, we should declare it https://review.openstack.org/34154 | 02:30 |
| *** dkliban_ has joined #openstack-infra | 02:33 | |
| *** CaptTofu has quit IRC | 02:34 | |
| *** CaptTofu has joined #openstack-infra | 02:34 | |
| mordred | Alex_Gaynor: once we've run it for a bit and feel that it's solid enough | 02:36 |
| mordred | zul: ping | 02:37 |
| mordred | zul: we've merged d2to1 and pbr which means that the d2to1.extern.six bundle is going away | 02:38 |
| mordred | zul: not urgently, but will be leaving | 02:38 |
| Alex_Gaynor | mordred: was just curious if there was a condition for "solid enough" | 02:38 |
| mordred | zul: I don't _think_ you were using it for anything other than pbr itself - but if you were, just wanted to give you a heads up | 02:38 |
| mordred | Alex_Gaynor: nope. I think it's "when sdague has watched it succeed for long enough that he feels doing the perm swap won't kill everyone" | 02:39 |
| *** CaptTofu has quit IRC | 02:39 | |
| mordred | Alex_Gaynor: but it's eagerly wanted :) | 02:39 |
| clarkb | for nova it was known issues fixed and next couple weeks will be quiet | 02:40 |
| Alex_Gaynor | mordred: is there a descipriont of each tempest run anywhere? | 02:40 |
| clarkb | also vishy being adventurous :) | 02:40 |
| mordred | Alex_Gaynor: yes! | 02:40 |
| mordred | Alex_Gaynor: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml shows the job names of everything zuul is going to trigger for a given purpose | 02:41 |
| mordred | https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml | 02:41 |
| mordred | contains the description of most of the jenkins devstack jobs | 02:41 |
| mordred | (all of the jobs are here: https://github.com/openstack-infra/config/tree/master/modules/openstack_project/files/jenkins_job_builder/config) | 02:41 |
| mordred | and devstack-gate itself is here: | 02:42 |
| mordred | https://github.com/openstack-infra/devstack-gate | 02:42 |
| Alex_Gaynor | thanks | 02:42 |
| mordred | with https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh and https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh being the most interesting bits for that question | 02:42 |
| * mordred throws WAY more information at Alex_Gaynor than he asked for, giggles | 02:42 | |
| Alex_Gaynor | heh | 02:44 |
| mordred | Alex_Gaynor: also, for your evening reading pleasure, http://ci.openstack.org is the complete documentation of how everything we do here hangs together | 02:48 |
| Alex_Gaynor | mordred: I believe I've read that, although it had a different sphinx theme when I did, which leads me to believe it may be different content | 02:48 |
| Alex_Gaynor | it was definitely at that domain though | 02:48 |
| mordred | Alex_Gaynor: jeblair did a complete rework of the content a little while ago, but I believe the theme change is orthogonal, so the content may or may not be similar | 02:49 |
| clarkb | the theme landed shortly after the rework | 02:51 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Add support for globbing in data files https://review.openstack.org/35730 | 02:55 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Add more documentation https://review.openstack.org/37553 | 02:55 |
| *** dkliban_ has quit IRC | 03:00 | |
| *** sarob has joined #openstack-infra | 03:01 | |
| *** sarob has quit IRC | 03:06 | |
| *** SpamapS_ is now known as SpamapS | 03:11 | |
| *** dkliban_ has joined #openstack-infra | 03:11 | |
| *** woodspa_ has quit IRC | 03:22 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Fix python 3.3 tests https://review.openstack.org/38084 | 03:25 |
| *** yaguang has quit IRC | 03:31 | |
| *** SergeyLukjanov has joined #openstack-infra | 03:33 | |
| lifeless | Alex_Gaynor: http://feedback.rackspace.com/forums/71021-product-feedback/suggestions/3882972-allow-upload-of-existing-vms | 03:42 |
| *** yaguang has joined #openstack-infra | 03:44 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Move d2to1 more into the source tree https://review.openstack.org/38079 | 03:46 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Remove the need to specify the pbr hook https://review.openstack.org/38080 | 03:47 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Replace entry_points console_scripts https://review.openstack.org/38000 | 03:48 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Do not assume the tests run as jenkins https://review.openstack.org/38071 | 03:48 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Ignore project creator while generating AUTHORS https://review.openstack.org/37624 | 03:56 |
| *** sarob has joined #openstack-infra | 04:02 | |
| fungi | pretty well given up on the wireless here at the hotel. it's sad to think that my wireless modem is more stable | 04:03 |
| clarkb | fungi: :( | 04:05 |
| *** dkliban_ has quit IRC | 04:06 | |
| clarkb | fwiw PSU has guest wireless that is free and mostly non sucky | 04:06 |
| *** dkliban_ has joined #openstack-infra | 04:06 | |
| *** sarob has quit IRC | 04:07 | |
| fungi | eh, this modem is actually not bad, performance-wise. it just always amuses me to be getting internet access via ppp again after so many years | 04:08 |
| *** s1rp has quit IRC | 04:13 | |
| *** arosen1 has joined #openstack-infra | 04:14 | |
| *** arosen has quit IRC | 04:18 | |
| *** yaguang has quit IRC | 04:30 | |
| *** s1rp has joined #openstack-infra | 04:37 | |
| *** dkliban_ has quit IRC | 04:43 | |
| *** koolhead17 has joined #openstack-infra | 04:46 | |
| *** yaguang has joined #openstack-infra | 04:52 | |
| *** Ryan_Lane has quit IRC | 04:57 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Test for IPv6 DSCP related issues. https://review.openstack.org/37745 | 05:16 |
| *** tian has joined #openstack-infra | 05:18 | |
| *** vogxn has joined #openstack-infra | 05:19 | |
| *** llu has joined #openstack-infra | 05:25 | |
| tian | hi,when I update patch set ,I got an error of jenkins "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset." | 05:25 |
| tian | and I Depend On the last patch | 05:27 |
| clarkb | tian can you link to the change? | 05:30 |
| clarkb | it will give me a concrete example of the behavior | 05:31 |
| *** tian_ has joined #openstack-infra | 05:33 | |
| *** tian has quit IRC | 05:33 | |
| llu | I think tian's patch is https://review.openstack.org/#/c/35767/ | 05:34 |
| *** dguitarbite has quit IRC | 05:35 | |
| clarkb | llu: thank you | 05:37 |
| clarkb | tian_: I just updated nova master and rebased your change atop that and it does indeed conflict | 05:37 |
| clarkb | tian_: you will need to do the same, resove the conflicts, then push the resulting commits back to gerrit | 05:38 |
| *** tian_ has quit IRC | 05:38 | |
| llu | clarkb: seems tian's patch is based on other 2 patches, https://review.openstack.org/#/c/35760/12, https://review.openstack.org/#/c/35759/12, does that means those 2 needs to be rebased first? | 05:40 |
| clarkb | llu: yes, it looks like https://review.openstack.org/#/c/35759/12 conflicts. However tian can just do that | 05:41 |
| clarkb | there is no reason to wait for shane if you are impatient and just want to do it | 05:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Pull toci into openstack-infra https://review.openstack.org/37432 | 05:41 |
| *** Ryan_Lane has joined #openstack-infra | 05:45 | |
| *** koolhead17 has quit IRC | 05:50 | |
| *** linuxmohan has joined #openstack-infra | 05:51 | |
| *** openstackgerrit has quit IRC | 05:54 | |
| *** openstackgerrit has joined #openstack-infra | 05:54 | |
| *** sarob has joined #openstack-infra | 06:03 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: We force installs via pip, we should declare it https://review.openstack.org/34154 | 06:04 |
| *** sarob has quit IRC | 06:07 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Conditionally install Python 3 develoment libs. https://review.openstack.org/38103 | 06:09 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Conditionally install Python 3 development libs. https://review.openstack.org/38103 | 06:15 |
| *** sdake has joined #openstack-infra | 06:16 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp behaviour https://review.openstack.org/37620 | 06:29 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp behaviour https://review.openstack.org/37620 | 06:30 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp behaviour https://review.openstack.org/37620 | 06:31 |
| *** SergeyLukjanov has quit IRC | 06:38 | |
| *** ianw has quit IRC | 06:42 | |
| *** ianw has joined #openstack-infra | 06:42 | |
| *** sdake has quit IRC | 06:52 | |
| *** yaguang has quit IRC | 07:00 | |
| *** sarob has joined #openstack-infra | 07:04 | |
| *** shanewang has joined #openstack-infra | 07:04 | |
| *** llu has left #openstack-infra | 07:07 | |
| *** sarob has quit IRC | 07:08 | |
| *** yaguang has joined #openstack-infra | 07:12 | |
| *** hashar has joined #openstack-infra | 07:20 | |
| *** shanewang1 has joined #openstack-infra | 07:26 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:27 | |
| *** shanewang has quit IRC | 07:30 | |
| ttx | yay travel. See you all on the other side. | 07:32 |
| hashar | have safe travelling! | 07:32 |
| fungi | see you when you get here, ttx | 07:34 |
| *** vogxn has quit IRC | 07:34 | |
| fungi | mordred: dhellmann_: py33 for pbr master is passing on precisepy3k-1.slave.o.o now | 07:35 |
| Alex_Gaynor | fungi: quick, and it to CI before someone breaks it | 07:35 |
| Alex_Gaynor | add* | 07:35 |
| fungi | mordred: dhellmann_: https://jenkins.openstack.org/job/fungi-node-tester/33/consoleText | 07:37 |
| fungi | Alex_Gaynor: well, we still need to get some reviews on https://review.openstack.org/38007 and https://review.openstack.org/38103 and merge them before i spin up a set of official slaves | 07:38 |
| Alex_Gaynor | unfortunately I'm not qualified to review either of those | 07:39 |
| *** shardy has joined #openstack-infra | 07:39 | |
| fungi | Alex_Gaynor: just as well, i wasn't qualified to write them | 07:41 |
| Alex_Gaynor | heh | 07:45 |
| *** Ryan_Lane has quit IRC | 07:46 | |
| *** jpich has joined #openstack-infra | 07:52 | |
| *** shanewang has joined #openstack-infra | 07:52 | |
| *** shanewang1 has left #openstack-infra | 07:54 | |
| *** jhesketh__ has joined #openstack-infra | 07:55 | |
| mikal | fungi: you got a minute to answer dumb questions? | 07:55 |
| fungi | mikal: maybe. about to drift off and get a few hours of sleep. what's the question? | 07:58 |
| mikal | Oh, I wanted to ask some CI advice | 07:59 |
| mikal | But its probably more than a couple of minutes worth | 07:59 |
| *** yaguang has quit IRC | 08:00 | |
| fungi | i guess it's unlikely you'll be at oscon | 08:00 |
| mikal | Yeah, I can't travel at the moment for family reasons | 08:00 |
| mikal | Its cool. I'll ping someone on IRC tomorrow morning if you need to sleep | 08:00 |
| *** afazekas has joined #openstack-infra | 08:00 | |
| fungi | i can always get you started, or follow up when i wake back up | 08:01 |
| mikal | Ok | 08:01 |
| mikal | So here's the thing | 08:01 |
| mikal | I have some python code which reads the gerrit event stream, and runs DB upgrade tests for nova migrations | 08:02 |
| mikal | It reports on failures, slow migrations, etc | 08:02 |
| mikal | Now jhesketh__ and I want to make it something solid and stable | 08:02 |
| mikal | But we're wondering if we should have gone down the jenkins route instead of something custom | 08:02 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added post-task publisher https://review.openstack.org/37924 | 08:02 |
| mikal | So I wanted to pick your brains about that... | 08:02 |
| fungi | mikal: it depends on whether your suite implements features that jenkins is well-engineered for | 08:03 |
| fungi | for example, dprince's smokestack doesn't use a jenkins | 08:04 |
| *** sarob has joined #openstack-infra | 08:04 | |
| mikal | Yeah, this is kind of like smokestack, except in python | 08:04 |
| fungi | at least i don't think he does | 08:04 |
| mikal | Which is how I got here mostly | 08:04 |
| *** jhesketh__ is now known as jhesketh | 08:04 | |
| mikal | The biggest constraint I think is that certain tests can only run on certain machines | 08:04 |
| mikal | (i.e. percona tests on a machine with percona not mysql) | 08:04 |
| mikal | And I need to be able to do markup on the logs from the test runs to make them easier to read | 08:05 |
| fungi | so, zuul is an interesting example of doing things like that using gearman. gearman workers register the jobs they're able to run | 08:06 |
| fungi | at least the specific jobs on specific machines thing | 08:06 |
| fungi | jenkins doesn't bring a lot of log markup features out of the box either, as far as i'm aware | 08:07 |
| mikal | Oh, interesting | 08:07 |
| mikal | Does zuul do any log markup? | 08:07 |
| mikal | It sounds like jenkins isn't a winner but that we should look at gearman? | 08:07 |
| fungi | no, sdague is working on a change to do some things like that through apache where the logs are served | 08:07 |
| fungi | the log markup stuff | 08:08 |
| mikal | Ahhh, interesting | 08:08 |
| mikal | He and I should talk | 08:08 |
| fungi | sorry for the lag in conversation. this wireless modem is very laggy | 08:08 |
| *** yaguang has joined #openstack-infra | 08:08 | |
| mikal | jhesketh was proposing that we could do markup in the browser with javascript | 08:08 |
| *** vogxn has joined #openstack-infra | 08:08 | |
| mikal | fungi: dude, no problem | 08:08 |
| mikal | At the moment I do markup by generating static HTML files on disk, which is horrible | 08:09 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Move Babel to test requires https://review.openstack.org/38114 | 08:09 |
| *** sarob has quit IRC | 08:09 | |
| fungi | browser-side js processing could be a little slow if these are huge logs | 08:09 |
| mikal | Yeah, mine are only a 1,000 lines per run | 08:09 |
| fungi | sdague's work is in https://review.openstack.org/37138 | 08:10 |
| mikal | Cool | 08:11 |
| mikal | I shall take a look | 08:11 |
| mikal | I'm also going to do some gearman research | 08:11 |
| fungi | i think we last talked about maybe switching it from python-cgi to mod_python | 08:11 |
| mikal | Thanks for your help | 08:11 |
| Alex_Gaynor | You probably don't want mod_python, it's deprecated and no longer supported. | 08:12 |
| Alex_Gaynor | mod_wsgi is the successor. | 08:12 |
| fungi | Alex_Gaynor: ahh, good to know. wsgi does seem like an okay fit as well | 08:12 |
| *** linuxmohan has quit IRC | 08:13 | |
| fungi | mikal: my pleasure. ask mordred, jeblair and clarkb the same questions tomorrow and you may get better answers to some of that too | 08:13 |
| mikal | Cool | 08:14 |
| fungi | er, later today i guess | 08:16 |
| BobBall | btw - any clue why some patches (e.g. https://review.openstack.org/#/c/35830/) have been approfved but not merged? | 08:21 |
| Alex_Gaynor | BobBall: I assume because the thing it depends on hasn't been merged | 08:21 |
| BobBall | d'oh | 08:22 |
| BobBall | sorry | 08:22 |
| BobBall | of course... | 08:22 |
| * BobBall is being thick this morning | 08:22 | |
| *** derekh has joined #openstack-infra | 08:24 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Allow hacking 0.6.0 and fix versions https://review.openstack.org/38119 | 08:29 |
| *** openstackstatus has quit IRC | 08:32 | |
| *** linuxmohan has joined #openstack-infra | 08:43 | |
| *** boris-42 has joined #openstack-infra | 08:45 | |
| *** jhesketh has quit IRC | 08:55 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Add .idea to .gitignore https://review.openstack.org/38121 | 08:55 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/38122 | 08:55 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 08:55 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 09:01 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 09:01 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Add .idea to .gitignore https://review.openstack.org/38121 | 09:01 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Allow hacking 0.6.0 and fix versions https://review.openstack.org/38119 | 09:01 |
| *** sarob has joined #openstack-infra | 09:04 | |
| *** sarob has quit IRC | 09:09 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Allow hacking 0.6.0 and fix versions https://review.openstack.org/38119 | 09:09 |
| *** johnthetubaguy has joined #openstack-infra | 09:10 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Add .idea to .gitignore https://review.openstack.org/38121 | 09:10 |
| jd__ | fungi: more info on this py33 host? | 09:11 |
| *** linuxmohan has quit IRC | 09:11 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 09:18 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 09:18 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 09:29 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 09:29 |
| *** rpodolyaka has joined #openstack-infra | 09:38 | |
| *** Guest66679 has quit IRC | 09:41 | |
| *** lillie has joined #openstack-infra | 09:41 | |
| *** lillie is now known as Guest63475 | 09:41 | |
| *** vogxn has quit IRC | 09:47 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 09:50 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 09:50 |
| *** koobs has joined #openstack-infra | 09:56 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 09:56 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 09:56 |
| *** sarob has joined #openstack-infra | 10:05 | |
| *** sarob has quit IRC | 10:10 | |
| lifeless | can I ask a favour - when moving repos, can we get a pingback, or a mail to openstack-dev or something, so that the info is disseminated rapidly? | 10:15 |
| lifeless | we had someone send a proposal to the old incubator location after the move, cause they didn't know. | 10:16 |
| *** vogxn has joined #openstack-infra | 10:23 | |
| *** stevebaker has left #openstack-infra | 10:24 | |
| *** fbo_away is now known as fbo | 10:26 | |
| *** stevebaker has joined #openstack-infra | 10:26 | |
| jd__ | dhellmann_: fwiw i'd be interested giving a spin on your infra VM, we have such a need for Hy and I'm struggling to build it | 10:32 |
| *** michchap_ has joined #openstack-infra | 10:37 | |
| *** michchap has quit IRC | 10:37 | |
| *** yaguang has quit IRC | 10:38 | |
| *** vogxn has quit IRC | 10:48 | |
| *** lcestari has joined #openstack-infra | 10:50 | |
| *** hashar has quit IRC | 10:56 | |
| *** hashar has joined #openstack-infra | 11:00 | |
| *** mestery_ has joined #openstack-infra | 11:02 | |
| *** mestery__ has joined #openstack-infra | 11:04 | |
| *** mestery has quit IRC | 11:05 | |
| *** sarob has joined #openstack-infra | 11:05 | |
| *** mestery_ has quit IRC | 11:07 | |
| *** sarob has quit IRC | 11:10 | |
| *** dkranz has quit IRC | 11:11 | |
| *** boris-42 has quit IRC | 11:30 | |
| *** plomakin has quit IRC | 11:37 | |
| sdague | mikal: so n-cpu is 35MB uncompressed, which means client side processing isn't going to be really possible | 11:39 |
| sdague | it already causes browsers a bit of heartburn just to be html | 11:40 |
| *** woodspa has joined #openstack-infra | 11:44 | |
| zul | mordred: ill have a look today | 11:44 |
| *** hashar has quit IRC | 11:48 | |
| *** boris-42 has joined #openstack-infra | 11:50 | |
| *** rfolco has joined #openstack-infra | 12:03 | |
| *** sarob has joined #openstack-infra | 12:06 | |
| *** dkranz has joined #openstack-infra | 12:07 | |
| *** hashar has joined #openstack-infra | 12:09 | |
| *** sarob has quit IRC | 12:10 | |
| *** dkliban_ has joined #openstack-infra | 12:16 | |
| *** CaptTofu has joined #openstack-infra | 12:16 | |
| *** woodspa has quit IRC | 12:21 | |
| *** woodspa_ has joined #openstack-infra | 12:21 | |
| *** openstackgerrit has quit IRC | 12:22 | |
| *** anteaya has joined #openstack-infra | 12:35 | |
| *** dprince has joined #openstack-infra | 12:43 | |
| *** cppcabrera has joined #openstack-infra | 12:45 | |
| *** zaro0508 has quit IRC | 12:47 | |
| *** dkliban_ has quit IRC | 12:47 | |
| *** jhesketh__ has joined #openstack-infra | 12:48 | |
| *** rpodolyaka has left #openstack-infra | 12:48 | |
| *** linuxmohan has joined #openstack-infra | 12:54 | |
| *** CaptTofu has quit IRC | 12:55 | |
| *** CaptTofu has joined #openstack-infra | 12:56 | |
| *** CaptTofu has quit IRC | 13:00 | |
| *** changbl_ has quit IRC | 13:02 | |
| jhesketh__ | Hi guys | 13:07 |
| jhesketh__ | could somebody please point me to where jenkins-jobs are stored? | 13:07 |
| jhesketh__ | there's a bunch of samples in https://github.com/openstack-infra/jenkins-job-builder but I can't seem to find the jobs that are prepared for the actual infra | 13:08 |
| * jhesketh__ hopes he makes sense | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:09 | |
| jhesketh__ | ah, found it (https://github.com/openstack-infra/config/tree/master/modules/openstack_project/files/jenkins_job_builder/config) | 13:09 |
| *** CaptTofu has joined #openstack-infra | 13:16 | |
| *** dhellmann_ is now known as dhellmann | 13:16 | |
| dhellmann | jd__: my "infra VM"? | 13:22 |
| *** mikal has quit IRC | 13:23 | |
| *** jhesketh__ has quit IRC | 13:24 | |
| jd__ | dhellmann: IIUC you were working on building a VM with Gerrit & cie | 13:24 |
| *** mikal has joined #openstack-infra | 13:25 | |
| dhellmann | jd__: ah, that | 13:27 |
| dhellmann | yes, that's still just in the "idea phase" but when it turns into a real thing, I'll make sure you know :-) | 13:27 |
| jd__ | dhellmann: thanks :) | 13:27 |
| *** NobodyCa1 is now known as NobodyCam | 13:29 | |
| *** mestery has joined #openstack-infra | 13:31 | |
| *** pentameter has joined #openstack-infra | 13:31 | |
| *** krtaylor has quit IRC | 13:34 | |
| *** mestery__ has quit IRC | 13:34 | |
| *** michchap_ has quit IRC | 13:39 | |
| *** michchap has joined #openstack-infra | 13:39 | |
| *** burt has joined #openstack-infra | 13:40 | |
| *** CaptTofu has quit IRC | 13:41 | |
| *** CaptTofu has joined #openstack-infra | 13:42 | |
| *** chuckieb has joined #openstack-infra | 13:43 | |
| *** vogxn has joined #openstack-infra | 13:46 | |
| *** CaptTofu has quit IRC | 13:46 | |
| *** CaptTofu has joined #openstack-infra | 13:52 | |
| *** prad has joined #openstack-infra | 13:52 | |
| sdague | any cormudgeon's about? | 13:54 |
| sdague | on last week's qa meeting we voted to take down our own mailing list, and use the main openstack-dev to keep us from getting too separated | 13:54 |
| sdague | n my ideal world I'd like emails sent to openstack-qa to actually go to openstack-dev and have [qa] automatically put into the subject line | 13:54 |
| *** linuxmohan has quit IRC | 13:57 | |
| *** dkliban_ has joined #openstack-infra | 13:59 | |
| *** dkliban_ is now known as dkliban | 14:00 | |
| *** yamahata_ has joined #openstack-infra | 14:01 | |
| *** linuxmohan has joined #openstack-infra | 14:02 | |
| dkranz | clarkb: I'm trying to add #oenstack-qa to irc logs bug on "git review" get this error. Any ideas? http://paste.openstack.org/show/41238/ | 14:07 |
| dkranz | clarkb: There is indeed no Change-Id there but I'm not sure what is supposed to put it there. | 14:08 |
| *** blamar has joined #openstack-infra | 14:09 | |
| *** _TheDodd_ has joined #openstack-infra | 14:10 | |
| anteaya | dkranz: line 4 says you are missing the change id | 14:10 |
| dkranz | anteaya: Yes, but what is supposed to put it there? | 14:11 |
| anteaya | which might have been auto generated by the attempt which created the output you pasted | 14:11 |
| anteaya | what happens when you try again | 14:11 |
| sdague | dkranz: git commit --amend, the git hook probably delayed in installation | 14:11 |
| anteaya | git review generates it | 14:11 |
| anteaya | but the calling of git review kicks it off if it isn't already there | 14:11 |
| anteaya | but that might cause it to fail the first time | 14:12 |
| *** rwsu_ is now known as rwsu | 14:12 | |
| dkranz | sdague: OK, that worked. Weird, this never happened to me before. | 14:12 |
| sdague | dkranz: only happens on the first git review in a repo | 14:12 |
| *** mrodden has joined #openstack-infra | 14:13 | |
| dkranz | sdague: Ah, ok. Perhaps I should add this to the wiki page https://wiki.openstack.org/wiki/Gerrit_Workflow | 14:14 |
| *** changbl_ has joined #openstack-infra | 14:15 | |
| *** michchap has quit IRC | 14:15 | |
| *** mestery_ has joined #openstack-infra | 14:19 | |
| *** mestery has quit IRC | 14:22 | |
| *** sarob has joined #openstack-infra | 14:37 | |
| *** krtaylor has joined #openstack-infra | 14:38 | |
| *** sarob has quit IRC | 14:42 | |
| *** markmcclain has joined #openstack-infra | 14:45 | |
| *** vogxn has quit IRC | 14:45 | |
| *** boris-42 has quit IRC | 14:46 | |
| *** weshay has joined #openstack-infra | 14:50 | |
| *** vijendar has joined #openstack-infra | 14:58 | |
| *** vijendar has left #openstack-infra | 14:58 | |
| *** vijendar has joined #openstack-infra | 14:58 | |
| *** alexpilotti has joined #openstack-infra | 14:59 | |
| alexpilotti | Hi guys I have a question related to git review, I asked already on -dev, burt helped but I still need to shed light on this: | 15:00 |
| alexpilotti | I have 1 change which depends on another one currently under review | 15:00 |
| alexpilotti | different branches and unrelated topics | 15:01 |
| alexpilotti | whatever I do, i get the dreaded "You have more than one commit that you are about to submit" | 15:01 |
| alexpilotti | what I did so far is: | 15:01 |
| alexpilotti | git fetch https://review.openstack.org/openstack/nova refs/changes/48/37948/6 && git checkout FETCH_HEAD | 15:01 |
| alexpilotti | checkout to my branch | 15:02 |
| alexpilotti | apply my changes and commit | 15:02 |
| alexpilotti | git review | 15:02 |
| alexpilotti | I'd expect that only the last commit gets sent for review | 15:03 |
| alexpilotti | but… no :-) | 15:03 |
| *** mrodden has quit IRC | 15:03 | |
| alexpilotti | If anybody could help it'd be great | 15:03 |
| jeblair | alexpilotti: unfortunately git-review isn't smart enough to know what gerrit is really going to do | 15:03 |
| alexpilotti | jeblair: I was hoping that by matching the change-id it would skip the first patch | 15:04 |
| jeblair | alexpilotti: so all it can do is say "i'm going to send these commits", for all of the commits you have that aren't merged yet. | 15:04 |
| jeblair | alexpilotti: so what should happen is that it sends both commits, and then gerrit sees that it already has the first commit, and just ignores it | 15:04 |
| burt | (which is what I said) | 15:05 |
| burt | :) | 15:05 |
| jeblair | alexpilotti: so only your new commit will do anything (but it will be based on the other one) | 15:05 |
| alexpilotti | jeblair: are you sure it doesn't overwrite the first one? | 15:05 |
| alexpilotti | burt: sorry: I didn't get that part in your answer | 15:05 |
| jeblair | alexpilotti: if the _commit sha_ is the same, then nothing will happen to the first one. | 15:05 |
| jeblair | alexpilotti: if the _change id_ is the same but the _commit sha_ is different, then it will update the first commit. | 15:06 |
| alexpilotti | jeblair: which is what happens with a commit --amend, makes sense | 15:06 |
| jeblair | alexpilotti: so if you want to be positive, run 'git log' and compare the git sha for for the 'other' commit and compare it to what gerrit displays | 15:06 |
| jeblair | alexpilotti: exactly | 15:06 |
| alexpilotti | jeblair: yep, just doing the git log now ;-) | 15:07 |
| alexpilotti | jeblair: on https://review.openstack.org is there a way to see the SHA? I see only the change-id | 15:09 |
| jeblair | alexpilotti: it's near the patchsets (the commit sha is displayed for each patchset, so check the latest) | 15:09 |
| jeblair | alexpilotti: to the right of "Patch Set N" | 15:10 |
| alexpilotti | jeblair: got it, tx! | 15:10 |
| BobBall | dprince: smokestack's JSON interface is down - so reviewday isn't refreshing | 15:11 |
| *** mgagne has joined #openstack-infra | 15:11 | |
| BobBall | dprince: correction... it only fails with the default request of 10,000 records... if you limit to fewer than that (e.g. 1000) then the query succeeds | 15:12 |
| dprince | BobBall: I'll have a look. | 15:13 |
| alexpilotti | jeblair: tx, it worked! :-) | 15:13 |
| *** mestery_ is now known as mestery | 15:14 | |
| BobBall | dprince: 2k records works as well - just doing a few tests... ;) | 15:14 |
| *** markmcclain has quit IRC | 15:17 | |
| *** rnirmal has joined #openstack-infra | 15:18 | |
| *** markmcclain has joined #openstack-infra | 15:19 | |
| *** vogxn has joined #openstack-infra | 15:20 | |
| *** mrodden has joined #openstack-infra | 15:22 | |
| *** sarob has joined #openstack-infra | 15:26 | |
| *** pcrews has joined #openstack-infra | 15:27 | |
| *** vijendar has quit IRC | 15:31 | |
| *** sarob has quit IRC | 15:32 | |
| Alex_Gaynor | mordred, dstufft: FYI: dnspython fixed their package, it's now hosted on pypi with no external scraping | 15:48 |
| *** markmcclain has quit IRC | 15:53 | |
| *** jgriffit1 is now known as jgriffith | 15:53 | |
| *** changbl_ has quit IRC | 15:55 | |
| *** changbl_ has joined #openstack-infra | 15:56 | |
| *** mjfork has joined #openstack-infra | 15:56 | |
| *** SergeyLukjanov has quit IRC | 15:59 | |
| SpamapS | who owns openstack/tripleo-incubator ? | 16:00 |
| *** vijendar has joined #openstack-infra | 16:00 | |
| mrodden | what was the source of the issue with xattr not installing correctly on friday? | 16:03 |
| mrodden | i think i'm running into that locally | 16:03 |
| *** ladquin has joined #openstack-infra | 16:04 | |
| *** afazekas has quit IRC | 16:05 | |
| clarkb | mrodden: new xattr requires libffi-dev to be installed so that it can compile cffi | 16:05 |
| clarkb | mrodden: (it also wouldn't build on some linuxes but xattr 0.7.1 fixed that) | 16:05 |
| mrodden | oh thats probably it | 16:06 |
| mrodden | missing ffi.h | 16:06 |
| mrodden | clarkb: thanks | 16:06 |
| *** atiwari has joined #openstack-infra | 16:06 | |
| *** atiwari has quit IRC | 16:07 | |
| clarkb | SpamapS: I have a couple minutes to fix your project permission things. can you give me the quick tl;dr so that I don't have to dig up details? | 16:07 |
| *** atiwari has joined #openstack-infra | 16:07 | |
| SpamapS | clarkb: sure, I believe the perms should be the same as tripleo-image-elements | 16:12 |
| SpamapS | clarkb: I don't know how to determine the exact names of those teams, you would probably find them faster. ;) | 16:13 |
| clarkb | SpamapS: what is your gerrit account id? There is a new tripleo-core group I will add you as a member to that so you can add who you want | 16:14 |
| clarkb | SpamapS: it can be found udner your gerrit settings page | 16:15 |
| clarkb | *account id number | 16:15 |
| *** vijendar has quit IRC | 16:15 | |
| jeblair | russellb: ext 7726 is a riot! | 16:17 |
| clarkb | SpamapS: nevermind sorted it out using the stackforge groujpe | 16:18 |
| clarkb | wow speeling is hard | 16:18 |
| clarkb | jeblair: good morning | 16:18 |
| jeblair | russellb: and 512-808-5750 seems to work :) | 16:18 |
| jeblair | clarkb: good morning! | 16:18 |
| SpamapS | clarkb: excellent thanks :) | 16:18 |
| clarkb | SpamapS: I have added lifeless, devananda, and yourself. You can add additional folks at https://review.openstack.org/#/admin/groups/190,members | 16:18 |
| *** hashar has quit IRC | 16:19 | |
| clarkb | jeblair: does the number of available d-g nodes look funny to you? | 16:20 |
| *** malini has joined #openstack-infra | 16:20 | |
| SpamapS | clarkb: perfect. Much appreciated. | 16:20 |
| * SpamapS approves a patch | 16:20 | |
| * SpamapS thinks there are very few actual funny numbers, but suspects there are either 42 or 69 d-g nodes ... | 16:20 | |
| clarkb | SpamapS: very near to 42.5 | 16:20 |
| SpamapS | possibly 3.. | 16:20 |
| *** mriedem has quit IRC | 16:22 | |
| clarkb | hmm looking at them via jenkins they seem ok | 16:22 |
| clarkb | but according to the zuul status page there is lots and lots of queueing and the number of nodes doesn't fall to 0 | 16:23 |
| dprince | BobBall: gearing up for some reviewday changes then? | 16:24 |
| *** sdake has joined #openstack-infra | 16:24 | |
| *** sdake has quit IRC | 16:24 | |
| *** sdake has joined #openstack-infra | 16:24 | |
| malini | hello!! | 16:24 |
| malini | what does an UNSTABLE means in the Jenkins gating job? | 16:24 |
| derekh | is there anybody around that can create apopulate a gerrit group ? https://bugs.launchpad.net/openstack-ci/+bug/1203689 | 16:24 |
| malini | I have this one https://review.openstack.org/#/c/38167/ that failed, but I dont get a page with the reason | 16:25 |
| jeblair | clarkb: the launch jobs seem to be doing the right thing | 16:25 |
| fungi | derekh: i can, just a sec | 16:25 |
| clarkb | jeblair: yeah, and jenkins does seem to be using most of the online slaves. | 16:25 |
| clarkb | I need to step out for a bit, but it looks like everything is correct and we are suffering from starvation and maybe graphite lag? | 16:26 |
| clarkb | jeblair: though the launch jobs do indicate >10 nodes available in each AZ however those numbers may be old if a bunch of jobs recently started after the launch jobs began | 16:26 |
| jeblair | clarkb: vmdb says there are 44 ready slaves | 16:27 |
| jeblair | clarkb: and 55 used | 16:28 |
| jeblair | clarkb: i suspect it has to do with the all-machine reap... | 16:30 |
| jeblair | clarkb: i think those machines may have been actually deleted but not removed from vm.db | 16:31 |
| *** andreaf has quit IRC | 16:31 | |
| jeblair | clarkb: so 2/3 of machines in the db don't actually exist | 16:31 |
| fungi | jeblair: clarkb: was the idea that the toci-core group would also include the openstack-ci-core | 16:33 |
| fungi | group | 16:33 |
| fungi | or just leave them to their own devices on that? | 16:34 |
| *** linuxmohan has quit IRC | 16:34 | |
| clarkb | jeblair: so manual curation may be necessary? | 16:35 |
| *** linuxmohan has joined #openstack-infra | 16:35 | |
| zaro | jeblair: i believe i figured out the jenkins logging issue. the logging should persist after jenkins restart. clarkb and i plan to look at it in a little bit. | 16:35 |
| jeblair | fungi: i believe they wanted us to be in it. I'm fine with that, under the understanding that we won't really be paying attention until they're closer to being able to run in the project infrastructure | 16:36 |
| jeblair | zaro: great! | 16:36 |
| jeblair | clarkb: yeah, i'll try to clean it up. | 16:36 |
| jeblair | clarkb: i also pushed https://review.openstack.org/38171 | 16:36 |
| fungi | okay, wfm | 16:36 |
| jeblair | going to fix gerritbot | 16:36 |
| *** markmcclain has joined #openstack-infra | 16:37 | |
| *** openstackgerrit has joined #openstack-infra | 16:37 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Set state to deleted when deleting https://review.openstack.org/38171 | 16:37 |
| jeblair | clarkb: there we go ^ | 16:38 |
| fungi | oof | 16:38 |
| *** mestery_ has joined #openstack-infra | 16:38 | |
| fungi | lgtm, not sure if you wanted to approve or wait for checks to come back (which would be a while on devstack-gate) | 16:39 |
| jeblair | clarkb: actually, i'll see if i can run the new check script (which isn't a job yet) and see if it automatically decides they should be deleted | 16:40 |
| jeblair | fungi: it's not urgent; the edge case it fixes only happens when we manually muck with the machines | 16:40 |
| BobBall | dprince: 'fraid not - just noticed it wasn't working! | 16:40 |
| BobBall | dprince: although I'm sure we can have a chat about changes that are needed | 16:41 |
| fungi | jeblair: ahh, okay. so that's not actually the cause of the current starvation i guess | 16:42 |
| dprince | BobBall: sure. if you've got ideas I'd love to hear them... | 16:42 |
| *** mestery has quit IRC | 16:42 | |
| jeblair | fungi: indirectly -- the current starvation is due to a single event from over the weekend (we manually deleted all the slaves) which this bug fixes; the event is not likely to recur. | 16:43 |
| *** pcrews has quit IRC | 16:43 | |
| jeblair | fungi: the fix to the starvation is that i'm running the check script which is validating that each server is still accessible (they are not) and deleting the ones that aren't. | 16:43 |
| jeblair | fungi, clarkb: that seems to be working (though somewhat slowly because it has to time out each server in turn) | 16:44 |
| fungi | got it. i had already forgotten about what transpired on saturday. it's like the flight wiped my memory of the first part of the weekend or something | 16:45 |
| *** krtaylor has quit IRC | 16:45 | |
| jeblair | fungi: temporal displacement will do that | 16:46 |
| BobBall | dprince: I've got more ideas for the SS website - filtering based on project / failures, showing latest failure reason in main iface, having links back to gerrit for easy navigation... | 16:46 |
| BobBall | dprince: but I've not even looked at where the SS website code is yet :D | 16:47 |
| dprince | BobBall: gotcha. cool. doesn't sound like there is anything too difficult there. | 16:48 |
| jeblair | fungi, clarkb: if you recheck the graph, you'll see the ready node count is starting to fall; yay! | 16:48 |
| dprince | BobBall: heads on the XenServer runs this morning. I hit a couple of issues. First was an issue with the latest PBR code (migrate.cfg wasn't getting deployed again). | 16:49 |
| jeblair | malini: it usually means that jenkins wasn't able to connect to the log server; if it's just one or a few jobs, it's probably a transient error and you can recheck | 16:49 |
| dprince | BobBall: At the same time I deployed a new RPM cache CGI script which broke things there as well. Both should be fixed now: I nailed PBR to an earlier revision and fixed the caching script. | 16:50 |
| dprince | BobBall: Still need to look into the PBR issue further and file a ticket though. | 16:51 |
| BobBall | Ahhh - understood. | 16:51 |
| *** jpich has quit IRC | 16:51 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:51 | |
| BobBall | When you've got details on the PBR issue - send us a line? (Mate and myself) | 16:51 |
| malini | jeblair: I think I had a bad commit message. I referred to a trello card (yeah we use trello) in the commit message. Interestingly, jenkins gave me a bad URL http://logs.openstack.org/67/38167/2/check/gate-marconi-pep8/955 with the unstable result | 16:53 |
| *** BobBall is now known as BobBall_Away | 16:54 | |
| *** mestery_ is now known as mestery | 16:54 | |
| jeblair | malini: yeah, the url it leaves doesn't work if jenkins can't copy the logs to the log server. | 16:55 |
| russellb | jeblair: awesome! | 16:55 |
| fungi | malini: the actual log for that run can be found at https://jenkins.openstack.org/job/gate-marconi-pep8/955/console in this case, for a little while still | 16:55 |
| russellb | jeblair: i had to log in and reload the configs earlier today. I haven't fixed up the puppet stuff to automatically reload yet ... | 16:55 |
| jeblair | malini: https://jenkins.openstack.org/job/gate-marconi-pep8/955/console | 16:56 |
| jeblair | malini: the pep8 test itself was fine | 16:56 |
| jeblair | oh fungi beat me to it | 16:56 |
| malini | thanks jeblair & fungi..its good to have tht url for the next time :) | 16:57 |
| fungi | malini: we only keep those around in jenkins for a fairly short period (a day i think) so it's only usually useful if you catch it right away | 16:57 |
| malini | fungi: tht's good to know | 16:58 |
| *** Ryan_Lane has joined #openstack-infra | 16:59 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Add python33 builder and job template https://review.openstack.org/37326 | 17:00 |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Enable python33 gate/check for pbr and oslo.config https://review.openstack.org/37426 | 17:00 |
| *** sarob has joined #openstack-infra | 17:00 | |
| mordred | Alex_Gaynor: w00t! | 17:02 |
| *** afazekas has joined #openstack-infra | 17:02 | |
| *** ^demon has joined #openstack-infra | 17:03 | |
| mordred | dprince: can you point me at the pbr issue you had? | 17:04 |
| mordred | dprince: and when you say latest, do you mean latest trunk or latest pypi? | 17:04 |
| *** boris-42 has joined #openstack-infra | 17:04 | |
| *** krtaylor has joined #openstack-infra | 17:04 | |
| dprince | mordred: ah yes. | 17:04 |
| dprince | mordred: I haven't filed a ticket yet. I only just discovered it this morning... | 17:04 |
| *** lbragstad has joined #openstack-infra | 17:05 | |
| dprince | mordred: A bunch of PBR commits when in today and one of those is the culprit for sure though. | 17:05 |
| mordred | dprince: k. cool. I'd love to get to the bottom of anything that's, you know, broken and stuff | 17:05 |
| *** linuxmohan has quit IRC | 17:05 | |
| dprince | mordred: The root cause/failure I saw was that migrate.cfg was missing again. | 17:05 |
| mordred | dprince: from the tarball? or from a setup.py install? | 17:05 |
| dprince | mordred: It was in the tarball though.... but not getting deployed for some reason. | 17:05 |
| *** linuxmohan has joined #openstack-infra | 17:06 | |
| dprince | mordred: setup.py install was the issue for sure. | 17:06 |
| mordred | dprince: fascinating. ok - that's great info - I'll go dig in to it | 17:06 |
| dprince | mordred: thanks sir. I'm actually having a bit of a rough time w/ PBR... which makes me think either: I need to report results back to Gerrit ahead of time (so we catch these) | 17:07 |
| dprince | mordred: or... I just stop using the latest and greated PBR. | 17:07 |
| *** derekh has quit IRC | 17:07 | |
| mordred | I'd love thing number 1 if possible - and also have been trying to add more combinations of things to integration.sh so that we don't break you | 17:07 |
| *** cody-somerville has joined #openstack-infra | 17:08 | |
| *** cody-somerville has quit IRC | 17:08 | |
| *** cody-somerville has joined #openstack-infra | 17:08 | |
| mordred | dprince: devstack is gating on trunk pbr now - but I believe it's using it differently from you, so I'd really like to make sure that we're not breaking you too | 17:08 |
| *** koolhead17 has joined #openstack-infra | 17:09 | |
| *** linuxmohan has quit IRC | 17:09 | |
| dprince | mordred: Yep. All a step in a right direction... | 17:09 |
| *** linuxmohan has joined #openstack-infra | 17:11 | |
| *** vogxn has quit IRC | 17:12 | |
| *** cppcabrera has left #openstack-infra | 17:15 | |
| *** cp16net is now known as cp16net|away | 17:16 | |
| sdague | jeblair: any thoughts on mailing list question I posted earlier? | 17:18 |
| *** sdake_ has joined #openstack-infra | 17:19 | |
| *** sdake_ has joined #openstack-infra | 17:19 | |
| mordred | sdague: was the question "who wants to buy me a hamburger?" | 17:21 |
| *** sgviking has quit IRC | 17:22 | |
| sdague | heh | 17:22 |
| sdague | mordred: no, it was on deprecating the openstack-qa list, and getting us over to openstack-dev | 17:23 |
| *** sgviking has joined #openstack-infra | 17:23 | |
| sdague | but what I was really hoping was that we could make openstack-qa an alias to openstack-dev which would auto add the subject tag | 17:23 |
| mordred | oh, that sounds too fancy for pre-coffee monty | 17:24 |
| sdague | what TZ you in to still be pre-coffee? | 17:26 |
| clarkb | sdague: the best TZ. PDT | 17:26 |
| clarkb | :) | 17:27 |
| sdague | yeh, but even there, it's 10:30, that's too late to be pre-coffee | 17:27 |
| sdague | :) | 17:27 |
| fungi | devstack ready count seems to be climbing again on the most recent datapoint. hoping that's a good sign | 17:28 |
| clarkb | yeah it looks much happier | 17:29 |
| fungi | dhellmann: jd__: you two probably need to merge efforts on https://review.openstack.org/38088 and https://review.openstack.org/38118 since one of them turns off distribute in tox and the other bumps the lower-bound on testrepository | 17:30 |
| fungi | but otherwise they're both the same patch | 17:31 |
| *** arborism has joined #openstack-infra | 17:34 | |
| dkranz | fungi: How can I find the reverify/rechecks that happened for a particular bug number? | 17:34 |
| *** blamar has quit IRC | 17:34 | |
| lbragstad | Hey all, just wondering if I could get another pair of eyes to look at this one here: https://review.openstack.org/#/c/34834/ It's saying its something with timeout around 40 minutes in and fails the build. If anyone has an idea feel free to let me know, I'm happy to help out. | 17:34 |
| fungi | dkranz: you can search for that bug number on http://status.openstack.org/rechecks | 17:34 |
| dkranz | fungi: Thx. | 17:35 |
| *** blamar has joined #openstack-infra | 17:35 | |
| fungi | dkranz: but that's only going to get you a list of which changes had at least one recheck and the total number of rechecks for that bug | 17:35 |
| fungi | hopefully that's enough for what you need | 17:35 |
| jeblair | sdague: i have not thought about it in detail, but i feel like it would be confusing and problematic. i feel like that's not a pattern that many people expect from mailing lists, and while it makes sense to us, it may not make sense to some people why emailing -qa causes their email to go to this other list. | 17:35 |
| jd__ | fungi: dhellmann: mine's better because it use the correct version of testrepository for py33 :-P | 17:36 |
| jeblair | sdague: i also think the mailman configuration for that wouldn't be easy. | 17:36 |
| *** nati_ueno has joined #openstack-infra | 17:36 | |
| sdague | jeblair: if you let openstack-dev consider openstack-qa an alias | 17:36 |
| sdague | and do it as an alias on the server | 17:37 |
| fungi | sdague: but you still need a special munger in exim to automagically add the [qa] tag in the subject header | 17:37 |
| jeblair | sdague: yeah, i think it would be technically possible with some work; i'm just wondering if it's worth explaining all of that to new users | 17:39 |
| sdague | you guys got procmail on there? | 17:39 |
| sdague | :0 fhw | 17:39 |
| sdague | * ^Subject: \/.* | 17:39 |
| sdague | | formail -I "Subject: [TAG] $MATCH" | 17:39 |
| dstufft | Alex_Gaynor: that makes me happy | 17:39 |
| jeblair | sdague: and why there's a -qa alias but not -nova -- or if we do that for all the topics, whether we can adequately explain that to a couple thousand people | 17:39 |
| fungi | lbragstad: how long does it take you to run the unit tests on that change? when it eventually finishes you can look at the testr stats to see which of the slowest tests are taking so long | 17:40 |
| sdague | jeblair: honestly, tagging is inconsistent enough, I bet you'd get much better tagging if there were aliases.... :) | 17:40 |
| jeblair | sdague: and don't forget that the To: header needs to be munged as well. or the Cc:; etc. | 17:40 |
| sdague | well we're already setting Reply-To | 17:42 |
| jeblair | sdague: or maybe you don't want to change the address headers; but then people on the -dev list will wonder why they are on the -qa list. | 17:42 |
| sdague | fair question on the To: / CC: though | 17:42 |
| sdague | anyway, it was a thought | 17:42 |
| clarkb | lbragstad: fungi: usually when that happens it is deadlock | 17:43 |
| jeblair | sdague: yeah, i'm going to keep thinking about it. :) i'm of two minds: "neat, i love doing cool things with email" and "i don't know if i want to try to explain that to new users". | 17:43 |
| clarkb | lbragstad: fungi: considering this is work around threads and removing eventlet that seems highly probable | 17:43 |
| *** cody-somerville has quit IRC | 17:44 | |
| sdague | jeblair: ok, cool. | 17:44 |
| sdague | well, I'll let you ponder more | 17:44 |
| jeblair | fungi, clarkb: yeah, the check scripts finished running and seemed to clean things up. i think that's ready to be a job now. | 17:45 |
| *** larsks has joined #openstack-infra | 17:47 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Gate jeepyb and openstack/requirements on pbr https://review.openstack.org/38176 | 17:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Use cgit server instead of github for everything https://review.openstack.org/38177 | 17:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Rename github-org to repo-group https://review.openstack.org/38178 | 17:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Clean up upstream cruft https://review.openstack.org/38179 | 17:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Add missing projects to pre-cache list https://review.openstack.org/38180 | 17:48 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Split projects list into its own file https://review.openstack.org/38181 | 17:48 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 17:48 |
| mordred | ok. patch bomb done | 17:48 |
| * mordred had a plane flight yesterday | 17:48 | |
| mordred | jeblair, pleia2 ^^ I jumped the gun on the git.o.o patches - but I wound up without internet on the plane without having prepared for such a thing, so I needed to find something I could hack on without context | 17:49 |
| *** cody-somerville has joined #openstack-infra | 17:49 | |
| jeblair | mordred: do you want me to review those? if you don't need early feedback on approach or something, i feel like they would be better as wip until ready for merging. | 17:51 |
| *** melwitt has joined #openstack-infra | 17:52 | |
| mordred | jeblair: yes. I'm going to go WIP the ones that are ahead of the game right now | 17:52 |
| mordred | jeblair: wip'd 3 of them. the others are fair game for now | 17:53 |
| jeblair | mordred: cool, thx | 17:54 |
| mordred | jeblair: the github -> git.o.o patch was a fun one to make though :) | 17:54 |
| pleia2 | if I could sync up with jeblair or fungi some time this week we can see about options we want on https://review.openstack.org/#/c/37794/ and see why the jeepyb script hasn't created a cgit config file yet | 17:54 |
| mordred | pleia2: ++ (I'll be happy to join that sync up) | 17:54 |
| pleia2 | cool | 17:54 |
| * pleia2 seeks refreshments | 17:55 | |
| mordred | pleia2: also - are we going to keep the cgit in the url? | 17:55 |
| pleia2 | mordred: I think so | 17:55 |
| mordred | pleia2: like, will it stay "git.o.o/cgit/openstack/nova" or will it be "git.o.o/openstack/nova" | 17:55 |
| mordred | ok. that patch might need updating | 17:55 |
| pleia2 | so when grabbing from http it's /cgit bit via git:// it's not (since that's not cgit) | 17:55 |
| jeblair | i'm in the CC lobby now | 17:56 |
| mordred | hrm. | 17:56 |
| *** changbl_ has quit IRC | 17:56 | |
| * mordred may want to talk about making /cgit disappear... but we can do that later - find refreshments! | 17:56 | |
| * jeblair notes git.kernel.org redirs to https://git.kernel.org/cgit/ | 17:56 | |
| mordred | yeah. I was going to just roll with it until "from http it's /cgit bit via git:// it's not" - which seems like a thing that people will get wrong when trying to type from memory | 17:57 |
| mordred | and doesn't really give us much benefit that I can see | 17:57 |
| mordred | BUT - I could be massively wrong | 17:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Periodically test devstack-node availability https://review.openstack.org/38186 | 17:58 |
| * mordred will now start making his way to the CC | 17:58 | |
| *** changbl has joined #openstack-infra | 17:58 | |
| clarkb | jeblair: russellb: pabelanger: is it fair game to approve asterisk changes or do you guys want to do them in a more controlled manner? | 17:58 |
| clarkb | (I am not sure if puppet is doing it all automagically yet) | 17:59 |
| russellb | it's fair game to approve IMO | 17:59 |
| jeblair | clarkb: and it is not auto-reloading | 17:59 |
| russellb | and i'm happy to review changes by others to asterisk configs if you'd like | 17:59 |
| russellb | but yeah, it's not auto-reloading ... | 17:59 |
| russellb | i tried to use notify => Service['asterisk'] or whatever, but didn't get it to work | 17:59 |
| clarkb | russellb: so if I approve https://review.openstack.org/#/c/38154/ do I need to do an asterisk service restart manually? (I can do that, just want to make sure I don't break anything. will calls be killed if I do that?) | 18:00 |
| russellb | pabelanger was going to work on some more in depth magic to only reload the module for the config file that changed, but dunno about status | 18:00 |
| russellb | yes, calls will die if you do that | 18:00 |
| russellb | sudo -u asterisk asterisk -rx "module reload" | 18:01 |
| russellb | will do it | 18:01 |
| *** cody-somerville has quit IRC | 18:01 | |
| russellb | service asterisk reload ... might do it too? | 18:01 |
| clarkb | russellb: and that performs an online reload without dropping calls? | 18:01 |
| russellb | correct | 18:01 |
| russellb | service asterisk reload is suppoted | 18:01 |
| russellb | and does the command i just said | 18:01 |
| russellb | so that's what we want | 18:01 |
| clarkb | cool I will go ahead and do this | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: pbx: Wait 1 second before playing a prompt https://review.openstack.org/38154 | 18:03 |
| *** sarob has quit IRC | 18:04 | |
| *** sarob has joined #openstack-infra | 18:04 | |
| *** sarob_ has joined #openstack-infra | 18:07 | |
| *** sarob has quit IRC | 18:07 | |
| *** sarob_ has quit IRC | 18:08 | |
| lbragstad | clarkb: I think you're right, on the Eventlet change. | 18:08 |
| lbragstad | clarkb: any idea what my next step might be? | 18:08 |
| *** sarob has joined #openstack-infra | 18:08 | |
| *** avtar has joined #openstack-infra | 18:10 | |
| *** datsun180b has joined #openstack-infra | 18:10 | |
| clarkb | lbragstad: you can run testr with the --until-failure option to have it run over and over until it fails. In this cause you wouldn't be looking for failure but deadlock | 18:10 |
| clarkb | lbragstad: source .tox/py27/bin/activate && testr run --parallel --until-failure | 18:10 |
| clarkb | you might also want to add --subunit to that and pipe the resulting stream to subunit2pyunit which should give you an idea of where things are hanging | 18:11 |
| lbragstad | ok | 18:11 |
| lbragstad | clarkb: thanks, giving that a shot now | 18:12 |
| anteaya | russelb: should I be able to connect with jitsi yet? I just tried and can't yet, am I doing something wrong? | 18:12 |
| russellb | anteaya: works for me. | 18:12 |
| *** cp16net|away is now known as cp16net | 18:12 | |
| anteaya | hmmm | 18:12 |
| russellb | anteaya: try pbx.openstack.org though ... | 18:12 |
| anteaya | okay | 18:12 |
| *** sarob_ has joined #openstack-infra | 18:12 | |
| * anteaya tries | 18:12 | |
| *** sarob has quit IRC | 18:13 | |
| russellb | literally dial "sip:s@pbx.openstack.org" | 18:13 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Support sip:conference@pbx.openstack.org https://review.openstack.org/38189 | 18:14 |
| *** sarob__ has joined #openstack-infra | 18:14 | |
| jeblair | ah, that looks nicer. :) | 18:14 |
| *** sarob__ has quit IRC | 18:14 | |
| russellb | heh, yeah | 18:14 |
| russellb | sip:pbx.openstack.org should work too, but looks weird | 18:14 |
| russellb | (and actually jitsi barfs on it) | 18:14 |
| Alex_Gaynor | Why is openstack looking to run its own pbx? | 18:16 |
| *** gyee has joined #openstack-infra | 18:16 | |
| russellb | jeblair: should I file a bug for SRV records? or? | 18:16 |
| russellb | Alex_Gaynor: self-hosted conference bridges is it right now | 18:17 |
| Alex_Gaynor | ah | 18:17 |
| jeblair | russellb: probably so; i think fungi was working on it, but his (our) attention may be sporadic this week | 18:18 |
| anteaya | russellb: I have an error: http://paste.openstack.org/show/41262/ is this an error with my client installation? | 18:18 |
| russellb | jeblair: ok, no rush, just want to be process friendly | 18:18 |
| fungi | jeblair: russellb: (i think) i was waiting for confirmation, but let me check my scrollback | 18:18 |
| fungi | i can add it now if we're ready | 18:19 |
| russellb | ah | 18:19 |
| *** cody-somerville has joined #openstack-infra | 18:19 | |
| *** cody-somerville has quit IRC | 18:19 | |
| *** cody-somerville has joined #openstack-infra | 18:19 | |
| russellb | anteaya: try "sip:s@pbx.openstack.org", what you have is missing the "s@" | 18:19 |
| anteaya | tried that too | 18:19 |
| russellb | anteaya: and you created a registrar-less SIP account first? | 18:19 |
| fungi | my workflow tends to be more like a stack, as much as i'd rather it were more like a pipe | 18:19 |
| russellb | fungi: same here ... | 18:20 |
| fungi | so that got pushed down by everything else which came up right after it :/ | 18:20 |
| russellb | and then i run out of stack space and start over | 18:20 |
| anteaya | russellb: I entered my username in the window that came up when I first started it, entered username in sip and nothing else | 18:20 |
| fungi | heh | 18:20 |
| russellb | anteaya: *shrug* don't know what to tell you | 18:20 |
| anteaya | does that create a registrar-less SIP account for me | 18:20 |
| anteaya | oh | 18:20 |
| *** malini has left #openstack-infra | 18:22 | |
| russellb | fungi: created https://bugs.launchpad.net/openstack-ci/+bug/1203835 to track it, won't let me assign it though | 18:23 |
| russellb | :) | 18:23 |
| fungi | russellb: i think i've discovered a bug with rackspace's dns interface | 18:25 |
| fungi | it tries to be too smart for its own good | 18:25 |
| clarkb | fungi: no way | 18:25 |
| russellb | nice | 18:25 |
| fungi | i created the tcp one and then tried to create the udp one... "The parent domain already has an SRV record that is using port 5060 on host pbx.openstack.org" | 18:25 |
| anteaya | when I go to jitsi > tools > options I see I have a registrarless SIP account | 18:25 |
| russellb | ha! | 18:25 |
| russellb | fungi: well if it only lets you have one, i'd just leave the UDP one | 18:25 |
| fungi | because i guess they never imagined that tcp and udp services could have the same port numbers | 18:25 |
| *** afazekas has quit IRC | 18:26 | |
| fungi | russellb: okay, i'll change the current tcp one to a udp one and open another wtf ticket with rackspace | 18:26 |
| russellb | awesome | 18:26 |
| *** bclifford has joined #openstack-infra | 18:26 | |
| russellb | yay, that + the last patch means you can call ... sip:conference@openstack.org | 18:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Add build job for config reference https://review.openstack.org/37010 | 18:28 |
| *** sarob has joined #openstack-infra | 18:28 | |
| jeblair | russellb: very intuitive! | 18:28 |
| *** sarob has quit IRC | 18:30 | |
| * clarkb tries the service asterisk reload | 18:30 | |
| *** sarob has joined #openstack-infra | 18:31 | |
| clarkb | and done. the prompt wait should be in place | 18:31 |
| *** sarob_ has quit IRC | 18:32 | |
| *** devananda has quit IRC | 18:33 | |
| jeblair | [2013-07-22 18:33:57.911] WARNING[27780][C-00000010]: channel.c:5081 ast_write: Codec mismatch on channel Message/ast_msg_queue setting write format to slin from unknown native formats (nothing) | 18:34 |
| jeblair | [2013-07-22 18:33:57.911] WARNING[27780][C-00000010]: channel.c:5309 set_format: Unable to find a codec translation path from (nothing) to (slin) | 18:34 |
| *** devananda has joined #openstack-infra | 18:34 | |
| jeblair | russellb: ^ that's being repeated when i connect to the console | 18:34 |
| russellb | eep | 18:34 |
| russellb | well that's not supposed to happen. | 18:35 |
| openstackgerrit | SlickNik proposed a change to openstack-infra/config: Added OpenStack docs jobs for Trove. https://review.openstack.org/38191 | 18:37 |
| russellb | anteaya: btw it appears you tried to create an account in jitsi that resides on pbx.openstack.org, which is not what you want | 18:38 |
| anteaya | oh okay, I am reading up on how to evaluate my account setting now | 18:38 |
| anteaya | I have never worked with sip before, so it will take me a bit to figure out what it wants | 18:38 |
| russellb | jeblair: there are 2 channels stuck. type "Bridge", which is related to ConfBridge(). I can't force them to go away so we have to restart asterisk. (I'll do it now ...) | 18:39 |
| jeblair | russellb: ok | 18:39 |
| jeblair | No such command 'Bridge' (type 'core show help Bridge' for other possible commands) | 18:39 |
| jeblair | that was not an imperative. | 18:40 |
| jeblair | :) | 18:40 |
| russellb | sorry ... I meant the channel type was Bridge | 18:40 |
| russellb | that was what was stuck | 18:40 |
| jeblair | i worked that out eventually. :) | 18:40 |
| russellb | if you looked at "core show channels" | 18:40 |
| russellb | calling in the conference over and over, can't make it happen, so i guess we'll have to watch out for how to trigger it | 18:40 |
| russellb | omg i just talked to jeblair with like ... voices | 18:42 |
| jeblair | :) i haven't used my voice in a while. i'm out of practice. | 18:43 |
| russellb | same here | 18:43 |
| clarkb | does this mean I should start bringing a headset to work? | 18:43 |
| russellb | bah, my desk phone isn't configured for URI dialing | 18:43 |
| jeblair | clarkb: yeah, and you can use it for "work". | 18:43 |
| clarkb | I could just put a sip client on my phone | 18:44 |
| * clarkb does this | 18:44 | |
| clarkb | anyone know of a good android client? | 18:44 |
| jeblair | clarkb: android. | 18:44 |
| sdague | clarkb: sipdroid is supposed to be decent | 18:44 |
| * jeblair uses "Phone" | 18:44 | |
| sdague | it's baked in now? | 18:44 |
| * clarkb tries the built in | 18:44 | |
| jeblair | menu -> internet call settings | 18:44 |
| *** vipul is now known as vipul-away | 18:45 | |
| *** vipul-away is now known as vipul | 18:45 | |
| jeblair | but i have a "provider" (my own asterisk server); i haven't tried providerless calling | 18:45 |
| mgagne | Are console logs from UNSTABLE jobs copied to logs.o.o ? | 18:45 |
| lbragstad | clarkb: tried the suggestion, still hangs and I tried with subunit but something went wrong in my terminal, like a translation issue | 18:46 |
| clarkb | mgagne: it depends on what caused the UNSTABLEness | 18:46 |
| russellb | we could create accounts for some people ... but ... that takes up the management overhead of this another notch | 18:46 |
| clarkb | mgagne: some unstable jobs are unstable because the log copying failed | 18:46 |
| *** SlickNik has left #openstack-infra | 18:46 | |
| *** SlickNik has joined #openstack-infra | 18:46 | |
| mgagne | clarkb: like this one? https://jenkins.openstack.org/job/gate-tempest-devstack-vm-postgres-full/34112/console | 18:46 |
| clarkb | russellb: I don't think that is necessary. Looks like i just need a contact with a sip address then I can call that | 18:46 |
| russellb | cool | 18:46 |
| clarkb | mgagne: yup | 18:46 |
| clarkb | lbragstad: are you doing testr run --subunit --parallel --until-failure | subunit2pyunit? | 18:47 |
| clarkb | the subunit2pyunit will convert the binary stream to something readable | 18:47 |
| jeblair | to extension 'conference' rejected because extension not found in context 'public'. | 18:48 |
| jeblair | hrm | 18:48 |
| jeblair | maybe that change hasn't shown up yet... | 18:49 |
| lbragstad | clarkb: thanks, rerunning | 18:49 |
| jeblair | ah, yeah, not approved yet | 18:49 |
| russellb | s/conference/s/ should work | 18:50 |
| russellb | or to dial a bridge directly 6000@ through 6999@ | 18:50 |
| lbragstad | clarkb: this was the latest, it's been stuck here for a while tests.unit.db.sqlalchemy.test_migrations.TestWalkVersions.test_migrate_up_with_data ... ok | 18:50 |
| fungi | jeblair: russellb: worth noting, when i try to add that second srv rr via the rackdns cli (after divining the odd syntax choice they went with for their srv record requests), i still get exactly the same error message | 18:50 |
| clarkb | lbragstad: ok, so the tricky bit is its actually the next test that is causing the problem | 18:50 |
| jeblair | fungi: yeah, was wondering if that would be any better | 18:51 |
| lbragstad | clarkb: gotcha | 18:51 |
| clarkb | lbragstad: when testr first runs it spits out the commands it is running for each test runner. these commands should include some /tmp files that list the test for each test runner | 18:51 |
| fungi | jeblair: clearly i was wondering that too, but no such luck | 18:51 |
| clarkb | lbragstad: you will want to find the temp file with test_migrate_up_with_data in it, and the next test in that list is the one causing problems | 18:51 |
| lbragstad | clarkb: cool, checking now | 18:51 |
| clarkb | hmm native android sip may require a username | 18:52 |
| jeblair | yeah, it seems to want an account | 18:52 |
| fungi | i don't suppose Alex_Gaynor has any stake in rackspace cloud dns | 18:53 |
| * clarkb tries sipdroid | 18:53 | |
| Alex_Gaynor | I know nothing about cloud dns. | 18:53 |
| fungi | Alex_Gaynor: no worries. i'll just pretend it's all your fault anyway | 18:54 |
| Alex_Gaynor | fungi: that's a reasonable way to proceed in many cases! | 18:54 |
| lbragstad | clarkb: tests.unit.db.sqlalchemy.test_migrations.TestWalkVersions.test_migrate_up_with_data | 18:54 |
| lbragstad | tests.unit.db.sqlalchemy.test_migrations.TestWalkVersions.test_walk_versions_all_default | 18:54 |
| lbragstad | so that should be the problem test there | 18:55 |
| clarkb | lbragstad: yup I think so | 18:55 |
| * fungi suspects infinite recursion in the walk | 18:55 | |
| Alex_Gaynor | fungi: if there's anyway I can of assistance though, let me know | 18:55 |
| fungi | Alex_Gaynor: nah, i'll let one of the fanatics get back to me on it | 18:55 |
| clarkb | lbragstad: you should be able to confirm by doing `testr run test_walk_versions_all_default` | 18:56 |
| clarkb | lbragstad: that should run only that test | 18:56 |
| fungi | Alex_Gaynor: though do you have any interest in dscp/qos problems with ipv6 in some parts of rackspace's cloud server environment? | 18:57 |
| Alex_Gaynor | fungi: There are almost no topics in all of computing that I'm less qualified to speak on than that. I am not an expert^Wparticularly familiar with networking. | 18:57 |
| lbragstad | clarkb: ok trying that quick https://github.com/openstack/oslo-incubator/blob/master/tests/unit/db/sqlalchemy/test_migrations.py#L343 | 18:57 |
| fungi | Alex_Gaynor: figured it was worth a shot anyway ;) thanks though! | 18:58 |
| anteaya | russellb: it said it was initiating call but then nothing else happened, I removed the config file and started again | 18:59 |
| clarkb | lbragstad: if that test run on its own works fine, then it is possible that a test running before it is causing the problem | 18:59 |
| *** vipul is now known as vipul-away | 19:00 | |
| anteaya | after creating a registrarless account you have to connect to another registrarless account, which I had done but obviously I had not done it properly | 19:00 |
| *** guitarzan has left #openstack-infra | 19:00 | |
| lbragstad | clarkb: I ran test_migrate_up_with_data just fine and it passed, but test_walk_versions_all_default still hung when just running that test. | 19:01 |
| lbragstad | clarkb: which seems to be consistent with the results. | 19:01 |
| mordred | clarkb: hey - you wanna dial in to a silly conference call? | 19:02 |
| *** nati_uen_ has joined #openstack-infra | 19:02 | |
| mordred | clarkb: jeblair reed and I are talking on it | 19:02 |
| *** sarob has quit IRC | 19:02 | |
| mordred | clarkb: but we're all also sitting in the same room | 19:02 |
| clarkb | mordred: I can't get sipdroid to work right now | 19:02 |
| clarkb | mordred: or is this a different call? | 19:02 |
| *** sarob has joined #openstack-infra | 19:02 | |
| mordred | clarkb: you could use your phone | 19:02 |
| clarkb | sure | 19:03 |
| mordred | clarkb: we're just talking on the asterisk server for the fun on it | 19:03 |
| mordred | of it | 19:03 |
| clarkb | oh I was going to grab lunch | 19:03 |
| clarkb | I can call in after :) | 19:03 |
| mordred | nah. don't worry about it | 19:03 |
| clarkb | ok /me grabs lunch | 19:03 |
| *** vipul-away is now known as vipul | 19:04 | |
| *** nati_ueno has quit IRC | 19:04 | |
| anteaya | btw if I hang up (the call never does get connected) and try to call again I get a failed transport error message | 19:06 |
| anteaya | if I create a new config, I can get the call initiated once (it doesn't connect) and then subsequenct calls with the same config get an error message | 19:06 |
| anteaya | mint 14 built on quantal 64bit | 19:07 |
| russellb | anteaya: thanks for the wiki update | 19:08 |
| anteaya | welcome | 19:08 |
| anteaya | I represent the noob that will make every mistake possible | 19:09 |
| *** reed has joined #openstack-infra | 19:09 | |
| jog0 | fungi: https://review.openstack.org/#/c/38012/ can you expand the commit msg | 19:10 |
| jog0 | see https://wiki.openstack.org/wiki/GitCommitMessages#Information_in_commit_messages | 19:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Set state to deleted when deleting https://review.openstack.org/38171 | 19:12 |
| fungi | jog0: you bet | 19:12 |
| jog0 | fungi: thanks | 19:13 |
| fungi | jog0: i was unsure whether describing the original problem was warranted since it's a revert and the original problem is thus described in the commit message of the change which is to be reverted, but i agree that more detail on why it's no longer needed is potentially helpful | 19:15 |
| *** arborism has quit IRC | 19:16 | |
| jog0 | fungi: yes describing why its not needed is useful | 19:16 |
| fungi | right away | 19:16 |
| jog0 | in fact I wasn't sure if the revert was its not needed or the fix broke something else | 19:16 |
| lbragstad | clarkb: I tried to confirm that it was just my patch by checking out master and running the tests. Seems to hang on master as well: unittest.loader.ModuleImportFailure.tests.unit.db.sqlalchemy.test_migrate ... FAIL which would be this module https://github.com/openstack/oslo-incubator/blob/master/tests/unit/db/sqlalchemy/test_migrate.py | 19:18 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Fixed some line-length and whitespace warnings. https://review.openstack.org/33206 | 19:19 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Fixed pyflakes issues. https://review.openstack.org/33208 | 19:19 |
| *** changbl has quit IRC | 19:19 | |
| *** dkliban has quit IRC | 19:22 | |
| fungi | jog0: the impetus is that it's blocking some other features which want newer jinja2, if memory serves | 19:22 |
| pabelanger | russellb: clarkb: Sorry funeral. Ya, I have some puppet magic I plan to work on this week | 19:23 |
| pabelanger | so, when a config is change, reload of modules happen | 19:23 |
| fungi | jog0: but "it's no longer necessary" should be enough, and the people who want to make use of the newer libs can lobby for that revert as they see fit | 19:23 |
| pabelanger | etc | 19:23 |
| *** changbl has joined #openstack-infra | 19:23 | |
| jog0 | fungi: thanks | 19:25 |
| fungi | np | 19:26 |
| mordred | fungi: would you be so kind as to: https://review.openstack.org/#/c/36923/ | 19:27 |
| dhellmann | mordred, fungi : I'm making notes about the steps to import WSME into stackforge. What's the story with CLA enforcement on stackforge projects? Can that be disabled for this one? | 19:28 |
| mordred | dhellmann: yes. CLA enforcement is totally optional | 19:29 |
| mordred | dhellmann: there's a setting in the acl file | 19:29 |
| dhellmann | cool, I think I asked that before but wanted to make sure | 19:29 |
| * dhellmann reads more closely and sees it | 19:29 | |
| mordred | dhellmann: btw - I'm just about set to make you a new pbr release - but we might have broken something else that dprince found, so I want to fix that first | 19:29 |
| dhellmann | :-S | 19:30 |
| *** vijendar has joined #openstack-infra | 19:30 | |
| *** cody-somerville has quit IRC | 19:31 | |
| *** nati_uen_ has quit IRC | 19:33 | |
| *** nati_ueno has joined #openstack-infra | 19:34 | |
| *** reed has quit IRC | 19:34 | |
| dhellmann | do we have python 2.5 installed on any of the jenkins slaves? | 19:38 |
| *** gyee has quit IRC | 19:38 | |
| *** chuckieb has quit IRC | 19:39 | |
| *** mriedem has joined #openstack-infra | 19:44 | |
| *** hashar has joined #openstack-infra | 19:44 | |
| *** bnemec_ is now known as bnemec | 19:52 | |
| *** derekh has joined #openstack-infra | 19:56 | |
| *** dprince has quit IRC | 19:59 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Add more documentation https://review.openstack.org/37553 | 20:01 |
| *** odyi has quit IRC | 20:01 | |
| *** odyi has joined #openstack-infra | 20:02 | |
| Ng | dstufft: hey, mordred suggested you would be a good person to talk to about a weird pip SSL error when using a proxy :) | 20:02 |
| dstufft | Ng: Sure | 20:03 |
| * clarkb is back | 20:03 | |
| Ng | dstufft: so this has come up in diskimage-builder (newly part of openstack) when it's running pip in a chroot, specifically "pip install os-apply-config". | 20:04 |
| jog0 | mordred: has your faster bin scripts been releases in a new pbr yet? | 20:04 |
| Ng | dstufft: the specific error is: Download error on https://pypi.python.org/simple/pbr/: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol -- Some packages may not be found! | 20:04 |
| jog0 | i.e. when can I test | 20:04 |
| clarkb | http://code.google.com/p/sipdroid/issues/detail?id=202 does that mean sipdroid isn't useful for calling into pbx.o.o? | 20:05 |
| Ng | dstufft: afaict, this actually bubbles up from setuptools when pip calls setup.py egg_info (to find dependencies) | 20:06 |
| lifeless | hi that; the bug bot in #openstack-meeting appears to not be | 20:06 |
| *** dkliban has joined #openstack-infra | 20:07 | |
| SpamapS | wasn't it just soren's uvirtbot? | 20:07 |
| clarkb | SpamapS: yes | 20:07 |
| Ng | dstufft: I have $http{,s}_proxy set, so I would have thought, from looking at the code, that the urllib2 opener chain should have had ProxyHandler as normal and detected that, but by VerifyingHTTPSHandler's https_open, the request object's has_proxy() returns False | 20:08 |
| dstufft | Ng: so you're doing pip install os-apply-config and you get an error trying to install pbr, that def does sound like setuptools during the setup_requires. | 20:09 |
| dstufft | lemme take a look at setuptools rq | 20:09 |
| soren | SpamapS: Hm? | 20:10 |
| SpamapS | soren: I can't seem to find uvirtbot | 20:11 |
| soren | SpamapS: It got really confused the other day about some SSL nonsense. Perhaps it fell over again. | 20:12 |
| * soren goes and looks | 20:12 | |
| *** SergeyLukjanov has quit IRC | 20:13 | |
| SpamapS | soren: ty | 20:13 |
| *** arborism has joined #openstack-infra | 20:14 | |
| *** johnthetubaguy has quit IRC | 20:14 | |
| *** linuxmohan has quit IRC | 20:21 | |
| dstufft | Ng: I hate urllib2 but if I'm reading this correctly it looks like setuptools lost proxy support | 20:23 |
| lbragstad | clarkb: found another issue that might be the reason why tests are failing with master, No module named migrate.changeset.constraint: which might be from sqlalchemy-migrate not being at 7.2 | 20:24 |
| Ng | dstufft: interesting, where are you seeing that? | 20:27 |
| *** boris-42 has quit IRC | 20:29 | |
| dstufft | Ng: https://bitbucket.org/pypa/setuptools/src/9938987d03379a4c129f31a437f5f6abb54d29ce/setuptools/ssl_support.py?at=default#cl-193 | 20:29 |
| dstufft | looks like VerifyingHTTPSHandler is trying to connect directly to the host/port | 20:30 |
| dhellmann | Alex_Gaynor: ping? | 20:30 |
| Alex_Gaynor | dhellmann: pong | 20:30 |
| dhellmann | are you working on setting up servers for testing code under pypy? | 20:30 |
| Alex_Gaynor | dhellmann: not yet, I want to get the swift tests passing (or mostly passing) first | 20:30 |
| dhellmann | ok | 20:31 |
| Alex_Gaynor | dhellmann: I've got at least 2 action items before I get there (one of which is gating now :P) | 20:31 |
| dhellmann | I'm looking into what it will take to move WSME to stackforge, and python 2.5 and pypy tests are one thing we'd lose if we did that | 20:31 |
| Alex_Gaynor | If you've got a use case for it, I'm happy to help getting it going sooner | 20:31 |
| * dhellmann realizes he can't count things | 20:31 | |
| Alex_Gaynor | Ok, so maybe we want to do that sooner? | 20:31 |
| dhellmann | yeah | 20:32 |
| dhellmann | also, omg, there are 67 environments described in the wsme tox.ini | 20:32 |
| Ng | dstufft: yeah I noticed that as well, what is a bit weird though is that in opener_for() just below that, the build_opener() call ought to leave ProxyHandler in the chain, but a debug print I added in VerifyingHTTPSHandler shows req.has_proxy() returns False, so I'm all kinds of confused at this point :) | 20:32 |
| Alex_Gaynor | hehe | 20:32 |
| dhellmann | holy cross-producs, batman | 20:32 |
| Ng | dstufft: (also that this seems to work normally outside the chroot) | 20:32 |
| Alex_Gaynor | dhellmann: gate all the things! | 20:32 |
| dhellmann | Alex_Gaynor: https://wiki.openstack.org/wiki/WSMEStackForgeTransition | 20:33 |
| dhellmann | one patch to wsme would consume all of the build slaves | 20:33 |
| dhellmann | so we'll have to batch them up :-) | 20:33 |
| Alex_Gaynor | or get some more build servers | 20:33 |
| dhellmann | yeah | 20:33 |
| *** vipul is now known as vipul-away | 20:34 | |
| *** vipul-away is now known as vipul | 20:34 | |
| Alex_Gaynor | dhellmann: let me tell you about this thing where you can elastically adjust the number of machines you need | 20:34 |
| dhellmann | heh | 20:35 |
| dhellmann | let me tell you about how long it takes to build a jenkins slave server | 20:36 |
| Alex_Gaynor | :[ | 20:37 |
| dhellmann | they're made with puppet instead of canned images | 20:37 |
| *** uvirtbot has joined #openstack-infra | 20:38 | |
| soren | uvirtbot: Meet SpamapS. | 20:38 |
| uvirtbot | soren: Error: "Meet" is not a valid command. | 20:38 |
| dstufft | Ng: is there a proxy outside of the chroot? | 20:40 |
| Ng | dstufft: yeah, it's running on another machine entirely | 20:40 |
| SpamapS | soren: many thanks for bringing your little supy back from the dead. :) | 20:41 |
| dstufft | Ng: Does the machine pip is running on have any access besides the proxy? (e.g. are you sure it's going through the proxy outside o the chroot) | 20:41 |
| anteaya | dhellmann: did you ever get an answer on the python2.5 running on -infra jenkins servers? I am curious about the answer to that question. | 20:41 |
| dhellmann | anteaya: I didn't see one, but disconnected not long after asking | 20:41 |
| anteaya | I didn't see one either | 20:41 |
| anteaya | but I am curious to know | 20:41 |
| Ng | dstufft: so, I just noticed that even though both host and chroot have the same python-setuptools package installed (0.6.28), the chroot also has a copy in /usr/local/ that's 0.9.something, which would explain the difference in behaviour if this ssl connection stuff is new | 20:42 |
| dhellmann | my guess is "no" | 20:42 |
| anteaya | my guess too | 20:42 |
| dstufft | Ng: oh | 20:42 |
| anteaya | I don't even know what file to look in for the answer | 20:42 |
| dstufft | yea 0.6 doesn't do SSL validation at all | 20:42 |
| dstufft | 0.7+ only | 20:42 |
| Ng | dstufft: right, well I guess I'm off to check for bugs on setuptools. Thanks for the help | 20:42 |
| dstufft | Ng: should be able to trigger it with easy_install I believe | 20:43 |
| lifeless | btw #action with the meeting bot isn't. Am I Using it wrong ? | 20:45 |
| *** koolhead17 has quit IRC | 20:45 | |
| mordred | lifeless: it's not putting actions in the log? | 20:46 |
| *** pcrews has joined #openstack-infra | 20:46 | |
| lifeless | indeed | 20:46 |
| lifeless | I do #action foo bar | 20:46 |
| lifeless | and it doesn't ack it like it used to. | 20:46 |
| lifeless | or is my expectation broken ? | 20:47 |
| mordred | lifeless: I do not believe #action has ever ack'd for us | 20:48 |
| lifeless | ah | 20:48 |
| lifeless | okiedoke | 20:48 |
| *** cody-somerville has joined #openstack-infra | 20:49 | |
| dhellmann | mordred, fungi, clarkb, jeblair: https://wiki.openstack.org/wiki/WSMEStackForgeTransition | 20:50 |
| *** melwitt has quit IRC | 20:55 | |
| *** melwitt1 has joined #openstack-infra | 20:55 | |
| *** vipul is now known as vipul-away | 20:57 | |
| *** vipul-away is now known as vipul | 20:57 | |
| *** melwitt1 is now known as melwitt | 20:57 | |
| *** gyee has joined #openstack-infra | 20:58 | |
| clarkb | I am going to restart jenkins-dev to try a log thing that zaro has found | 20:59 |
| *** woodspa_ has quit IRC | 21:04 | |
| dkranz | clarkb: logs.openstack.org seems to hang. That new? | 21:05 |
| clarkb | dkranz: are you trying to open the root dir? that will be slow because we have ~32k entries in that dir | 21:07 |
| clarkb | dkranz: last week we changed it to use two digit prefix dirs to make listing of directories faster | 21:07 |
| dkranz | clarkb: Ahh. So what is the formula for getting from a jenkins job url to the log url? | 21:08 |
| clarkb | dkranz: but it will still be slow until those 32k ish entries get old and are deleted | 21:08 |
| clarkb | dkranz: for tests run since friday it is http://logs.openstack.org/$PREFIX/$CHANGE/$PATCHSET/$QUEUE/$JOB_NAME/$RUN_NUMBER | 21:09 |
| clarkb | dkranz: where prefix is the last two digits of the change number | 21:09 |
| dkranz | clarkb: What about for a periodic job? | 21:09 |
| *** lcestari has quit IRC | 21:10 | |
| *** krtaylor has quit IRC | 21:10 | |
| clarkb | dkranz: those go under /periodic. I think opening /periodic should be fast neough | 21:10 |
| dkranz | clarkb: Great! | 21:10 |
| clarkb | dhellmann: python 3.2 and pypy are in theory things we can do fairly easily, assuming we don't have to do too much crazy pip stuff | 21:11 |
| *** sarob has quit IRC | 21:11 | |
| *** arborism has quit IRC | 21:11 | |
| dhellmann | clarkb: haha, "crazy pip stuff" | 21:11 |
| dhellmann | :-) | 21:11 |
| clarkb | dhellmann: I think we have found the tricky part of doing different pythons is that installing things globally for them is a PITA | 21:11 |
| clarkb | dhellmann: in theory tox makes this a non issue though | 21:11 |
| *** sarob has joined #openstack-infra | 21:11 | |
| dstufft | in thory pip works in python 3.2 and pypy just fine | 21:12 |
| *** cp16net is now known as cp16net|away | 21:12 | |
| dstufft | I mean it's tested and stuff | 21:12 |
| dstufft | could be broken anyways D: | 21:12 |
| *** HenryG has quit IRC | 21:12 | |
| clarkb | dstufft: pip itself isn't the issue. its you have to make pip run in those envs | 21:12 |
| clarkb | dstufft: puppet doesn't make that easy | 21:12 |
| dstufft | oh | 21:12 |
| dstufft | I don't use puppet so idk | 21:12 |
| *** HenryG has joined #openstack-infra | 21:12 | |
| * dstufft uses Chef, has a friend at opscode so free tech support ftw | 21:13 | |
| Alex_Gaynor | clarkb: for pypy the only bit is we might want a newer ppa than what's in ubuntu's stable repos, besides that a recent pip should work fine with it | 21:13 |
| dhellmann | clarkb: I guess we'd need to add all of the requirements to the pypi mirror, too | 21:13 |
| clarkb | Alex_Gaynor: dstufft: yeah pip itself isn't a problem. Convincing the mechanism that manages our hosts to have pip install package foo under python env bar is the hard part | 21:14 |
| * mordred sees chef in scrollback ... reads | 21:14 | |
| clarkb | I believe we have solved this with a different pip for each environment which is derpy if we want to have >1 env per slave | 21:14 |
| *** cp16net|away is now known as cp16net | 21:14 | |
| dhellmann | clarkb: we don't just use the pip from the env tox makes for us? | 21:14 |
| clarkb | dhellmann: we do when running tests, but we rely on pip to install tox :) | 21:15 |
| mordred | well... | 21:15 |
| clarkb | dhellmann: and there was some weird cross contamination on oneiric hosts using the python 2.7 tox | 21:15 |
| mordred | a) we want to stop installing stuff via pip on our servers | 21:15 |
| clarkb | mordred: we do? | 21:15 |
| dhellmann | clarkb: ouch, I think you broke my head ;-) | 21:15 |
| mordred | yes | 21:15 |
| mordred | because of all of the problems you were just talking about | 21:16 |
| clarkb | dhellmann: python2.6 jobs ran under python2.7 tox and well sometimes you imported stuff from python2.7 :/ | 21:16 |
| *** sarob has quit IRC | 21:16 | |
| Alex_Gaynor | you should be able to isntall tox into any python | 21:16 |
| Alex_Gaynor | it doesn't matter what hte host python is | 21:16 |
| jeblair | dhellmann: feedback on wiki: i don't anticipate a problem if you want to fix run-tox args (indeed, thanks!) | 21:16 |
| mordred | Alex_Gaynor: but it does | 21:16 |
| clarkb | in theory tox makes that work, but tox isn't perfect | 21:16 |
| dhellmann | clarkb: yep | 21:16 |
| mordred | Alex_Gaynor: because it runs some things outside of its venv | 21:16 |
| Alex_Gaynor | mordred: orly? | 21:16 |
| Alex_Gaynor | grumble grumble, have you tried filing a bug? | 21:16 |
| jeblair | clarkb: yeah, we decided that in one of the infra meetings when we were dealing with py33 | 21:16 |
| mordred | Alex_Gaynor: I made a patch that sohudl be in the next tox | 21:17 |
| Alex_Gaynor | mordred: oh great | 21:17 |
| mordred | that will allow you to configure tox to use setup.py develop | 21:17 |
| mordred | instead of sdist + install | 21:17 |
| mordred | and that develop will run in the context of the venv | 21:17 |
| mordred | so - basically, once the next versions of pip, venv and tox all release | 21:17 |
| mordred | several of these thigns shoudl get better | 21:17 |
| mordred | then we shoud make an apt repo thing for our infra systems | 21:18 |
| mordred | so that we can stop doing things with puppet's broken pip support | 21:18 |
| dhellmann | jeblair: I'll make that change regardless because I want it for running the docs jobs | 21:18 |
| jeblair | dhellmann: i'm not sure how i feel about the 67 envs issue. we have knobs we can tune: run 67 jobs, run 1 job that runs 67 commands, anything in between those 2.... | 21:18 |
| clarkb | jeblair: I knew we didn't want to do multiple pythons with pip (for the reasons I described above among other things) but I didn't realize we were wanting to go 100% native package for all things python | 21:18 |
| dhellmann | jeblair: I thought maybe 1 job per python version, which then runs several builders to handle the matrix | 21:18 |
| jeblair | dhellmann: we can batch them in zuul so that 1 job must complete before the second starts ... | 21:19 |
| mordred | clarkb: for infra systems - we still obviously want to use pip/venv for running jobs | 21:19 |
| jeblair | mordred: oh yes indeed | 21:19 |
| mordred | clarkb: at least, we've talked about it - and about making some jobs that auto-build debs/rpms for projects that we manage | 21:20 |
| dhellmann | jeblair: would batching them in zuul still use different slaves? | 21:20 |
| jeblair | dhellmann: yes | 21:20 |
| dhellmann | I'm worried about consuming all of the jenkins servers | 21:20 |
| jeblair | dhellmann: but one at a time | 21:20 |
| mordred | I think we could also add support for a few key things in our apt/yum repo for key things we don't control - such as tox, venv and pip | 21:20 |
| jeblair | dhellmann: have an idea how long the jobs run? | 21:20 |
| dhellmann | jeblair: let me run them and see :-) | 21:21 |
| jeblair | dhellman: i feel like if they are short, then it may not be worth worrying about starvation | 21:21 |
| dhellmann | jeblair: how long does it take to make a new jenkins slave? I thought that was time consuming? is that just for the devstack gate servers? | 21:22 |
| jeblair | dhellmann: but that's just an early initial thought. :) | 21:22 |
| clarkb | mordred: I feel that deploying things like zuul via debs/rpms adds a lot of unnecessary work | 21:22 |
| mordred | clarkb: why? | 21:22 |
| jeblair | clarkb: hashar does it. :) | 21:23 |
| clarkb | mordred: because you add another layer on top of the existing stuff where things can break | 21:23 |
| hashar | hello :) | 21:23 |
| *** mriedem has quit IRC | 21:23 | |
| clarkb | and hashar does have problems when new dependencies are used | 21:23 |
| hashar | well I have problems packaging them cause I am both a python and a debian noobie :-] | 21:23 |
| dhellmann | jeblair: 1.5 minutes for the py27 tests of wsme on my laptop with rotating storage | 21:24 |
| hashar | but I know the people able to assist in those tasks | 21:24 |
| *** pcrews has quit IRC | 21:24 | |
| jeblair | i think it's worth doing to help manage our production systems better | 21:24 |
| jeblair | and we can collaborate with hashar | 21:24 |
| hashar | clarkb: Wikimedia does deploy code with debian package. makes it easy to track dependencies and do downgrade (when the package is well written). Also that is a very nice way to easily figure out what version is installed | 21:24 |
| *** pcrews has joined #openstack-infra | 21:25 | |
| hashar | I filled a bunch of bugs in wikimedia bug tracker to get Zuul dependencies packaged. Apparently Zigo / OS already packaged most of them (like d2to1 , pbr). I think the python gear module is missing | 21:25 |
| hashar | Zuul 1.2 (aka pre gear) has been packaged by pabelanger on github. I guess I will have a look at the package again whenever I migrate to gear | 21:26 |
| jeblair | dhellmann: so our unit tests slaves are launched by hand; it's easy but takes a few human-minutes to do | 21:26 |
| mordred | clarkb: I don't think it will be any harder than what we do now | 21:26 |
| clarkb | hashar: jeblair mordred I think native packages do come with a lot of nice benefits including not sucking hard with puppet | 21:26 |
| dhellmann | jeblair: oh, so they're not rebuilt each time a job runs? | 21:26 |
| jeblair | clarkb: +1 | 21:26 |
| jeblair | dhellmann: that's only the devstack slaves | 21:27 |
| *** vipul is now known as vipul-away | 21:27 | |
| dhellmann | jeblair: ok, that's where I was confused, I thought that applied to everything | 21:27 |
| jeblair | dhellmann: we're not opposed to doing that across the board | 21:27 |
| clarkb | I just haven't convinced myself that the additional work is worth that benefit | 21:27 |
| mordred | clarkb: if we can auto-build them in a sensible manner | 21:27 |
| mordred | clarkb: I don't think it'll be extra work | 21:27 |
| jeblair | dhellmann: we need to either (a) get jclouds working or (b) make devstack-gate more generalized | 21:27 |
| mordred | I do NOT think that we need human-curated manual-released-like-debian/ubuntu does it packages | 21:27 |
| mordred | I think we need simple debs that install the python code | 21:27 |
| mordred | and I thnk that's almost 100% automatable | 21:28 |
| clarkb | mordred: it is whenever you have new dependencies and now yo uahve to track your dependencies and for each of those do packaging. all of a sudden you become package maintainer for a lot of stuff | 21:28 |
| mordred | only non-automatic thing will be dependency mapping | 21:28 |
| dhellmann | mordred: fpm is pretty good at making debs from setup.py | 21:28 |
| mordred | dhellmann: oh yeah? great! | 21:28 |
| jeblair | dhellmann: i think our favored approach is (b), but needs some more thought around it because the current method of making them single-use is kinda clunky | 21:28 |
| mordred | it would be in ruby | 21:28 |
| jeblair | dhellmann: clarkb may have thoughts on current feasibility of (a) | 21:28 |
| jeblair | (jclouds) | 21:28 |
| dhellmann | mordred: yeah, it may take a couple of tries to get the arguments just right | 21:28 |
| * mordred wants b | 21:28 | |
| mordred | dhellmann: cool | 21:29 |
| clarkb | b++ | 21:29 |
| * mordred started doing some hacking in that direction on the plane | 21:29 | |
| jeblair | i think the mechanism for marking slaves as in-use and pending-deletion needs to be moved outside of jenkins. maybe something triggered by zuul. | 21:29 |
| jeblair | mordred: oh? | 21:30 |
| mordred | it's not ready for coment yet - need to hack a few iterations to have better thoughts | 21:30 |
| *** vipul-away is now known as vipul | 21:30 | |
| mordred | but I'll try to have some sketches up that we can talk about and make fun of | 21:30 |
| jeblair | mordred: can you file/claim a bug? | 21:30 |
| mordred | jeblair: yeah | 21:30 |
| hashar | clarkb: usually we use the upstream debian packages, sometime we get our own patch there and sometime we heavily tweak them for our needs (though that last case is rare). | 21:32 |
| *** linuxmohan has joined #openstack-infra | 21:32 | |
| hashar | clarkb: also we don't install python modules from pypi. We need a package for every single module we need | 21:32 |
| clarkb | hashar: which is what we would need as well | 21:32 |
| clarkb | and that is where I see the majority of the extra work potentially happening | 21:33 |
| clarkb | as our dependencies are somewhat orthogonal to the work zigo is doing (though there is a fair bit of overlap) | 21:33 |
| hashar | luckily the Debian python packaging community is very active and helpful | 21:33 |
| hashar | and we get a bunch of sponsor to help get the packages included | 21:33 |
| mordred | https://github.com/jordansissel/fpm | 21:34 |
| hashar | mordred: the packages fpm produce are not up to par with Debian quality requirements :/ | 21:34 |
| hashar | mordred: though that might give a good start | 21:35 |
| hashar | clarkb: as I understand it Zigo is packaging the soft openstack is producing and is not really involved in the -infra stuff | 21:35 |
| clarkb | hashar: correct | 21:35 |
| hashar | clarkb: while pabelanger handle most of the -infra packaging | 21:35 |
| mordred | hashar: yes. we don't need them to be up to par with debian quality requirements | 21:35 |
| mordred | I don't need policy compliance for install zuul onto our own server | 21:36 |
| mordred | I need the python files that python setup.py install would install to wind up in the same place | 21:36 |
| hashar | mordred: if you get them at that level, that benefit the whole OSS / python community. It get included in Debian / Ubuntu and other people (such as Wikimedia) can benefit from them :) | 21:36 |
| mordred | hashar: totally, but these are different concerns | 21:36 |
| hashar | yup | 21:36 |
| jeblair | if someone wants to make them better and match debian policy, even better. | 21:36 |
| mordred | for instance, I do NOT want zuul.deb to control starting and stopping the zuul service | 21:36 |
| hashar | the thing is that most of the work is the actual packaging, the compliance with debian is not that much more so it might as well get done along the way | 21:37 |
| mordred | so doing the work to put service config into zuul.deb would be a waste of my time | 21:37 |
| *** linuxmohan has quit IRC | 21:37 | |
| hashar | why don't you want puppet to restart your zuul service hehe :-]]]]]] | 21:37 |
| jeblair | mordred: but maybe not someone else's, so +1 free software. :) | 21:37 |
| mordred | hashar: exactly - I want PUPPET to restart it... I _don't_ want dpkg | 21:37 |
| mordred | jeblair: ++ | 21:37 |
| mordred | yes - I'm totally on board for other people doing packaging work for these | 21:38 |
| mordred | but also, some we run from trunk | 21:38 |
| mordred | so I need fully-automatic package building and publication | 21:38 |
| mordred | which is also a very-bad-thing in debian land | 21:38 |
| hashar | one of our volunteer has been working on having Jenkins to build the packages for us (on each patchset submission and post merge) | 21:38 |
| mordred | BUT - I could see us having a packaging repo that we collaborate on with debian developers | 21:38 |
| mordred | hashar: I've got scripts to do that and inject into a reprepro ... need to be altered slightly | 21:39 |
| dhellmann | mordred: "fpm -t deb -s python ." in oslo.config built a package for /usr/local/lib/python2.7 -- is that what you'd want? | 21:39 |
| mordred | repo-per-git-branch etc | 21:39 |
| hashar | http://jenkins-debian-glue.org/ you get the packages build in a vm, copy the artifact to some place which happens to be a reprepro repository and then you can apt-get update and install the build package for you | 21:39 |
| mordred | dhellmann: that would be fine | 21:39 |
| dhellmann | mordred: obviously for something other than oslo.config, that's just what I had handy :-) | 21:39 |
| cody-somerville | It's interesting. There used to be a lot of discussion and buzz around Ubuntu/launchpad/etc. building and consuming from upstream trunks. I don't think it ever fully materialized sadly. | 21:39 |
| hashar | mordred: I want to get Wikimedia debian package build with jenkins by october :) | 21:40 |
| mordred | cody-somerville: it got caught up in the cross-purposes between upstream and distro policy | 21:40 |
| clarkb | dhellmann: does it package all of the dependencies for you too? or do you need to recursively run fpm? | 21:40 |
| dhellmann | clarkb: it does not package all of the dependencies, however if you have them all in a virtualenv you can use a different mode to make a package out of that | 21:41 |
| mordred | most of the deps are probably in distros already | 21:41 |
| dhellmann | clarkb: and it does add dependencies to the deb metadata | 21:41 |
| dhellmann | though that can probably be disabled, I'm not sure it's dependency name guessing algorithm is something I would trust | 21:41 |
| *** johnthetubaguy has joined #openstack-infra | 21:42 | |
| mordred | so - seriously though - simple does-nothing-but-install-python-files packages take about 10 minutes to make | 21:42 |
| jeblair | dhellmann: how many wsme tox env's per-platform (eg, how many python 2.7 tox envs?) | 21:42 |
| jeblair | dhellmann: i count ~22 for py27 in tox.ini (if my simple search is correct) | 21:43 |
| jeblair | dhellmann: so other platforms must have fewer variants, since 22*4 != 67 | 21:43 |
| clarkb | BobBall_Away: you still away? I would like to merge your meetbot changes and was wondering what sort of testing you had done? | 21:43 |
| dhellmann | jeblair: I came up with 22 as well | 21:43 |
| dhellmann | jeblair: yeah, some of them aren't python versions, they're for alternate web frameworks | 21:44 |
| clarkb | BobBall_Away: I can stand up a meetbot to test, but figured you had probably already done all of this since you were patching in part the tests | 21:44 |
| dhellmann | mordred: add --no-depends to that command line to have the package not declare its dependencies | 21:44 |
| jeblair | dhellmann: at any rate, with our current slave levels, that would consume all of our slaves for 1.5 minutes, and then 1/3 of them for a further 1.5 mins. | 21:44 |
| mordred | that will not be noticed | 21:45 |
| dhellmann | jeblair: I don't know if that's a good thing or not. | 21:45 |
| dhellmann | no? | 21:45 |
| dhellmann | ok | 21:45 |
| jeblair | dhellmann: that doesn't sound like the _worst_ thing in the world; though in some situtations, we may notice. | 21:45 |
| *** dkliban has quit IRC | 21:45 | |
| jeblair | dhellmann: i mean, if someone slings up a 10 change series, that's something else. :) | 21:45 |
| dhellmann | jeblair: I don't expect the project to have a high number of patches, although I could be wrong since the point of this is to increase contributions :-) | 21:46 |
| dhellmann | jeblair: oh, man, yeah | 21:46 |
| jeblair | dhellmann: i think i'd say start with letting other requirements drive the decisions -- | 21:46 |
| mordred | clarkb: ping: https://review.openstack.org/#/c/36923/ ? | 21:47 |
| jeblair | dhellmann: like does it make sense for jenkins/zuul to report back 67 different jobs/links in gerrit, or would it make more sense to report back one for each platform | 21:47 |
| *** dkranz has quit IRC | 21:47 | |
| dhellmann | jeblair: good point | 21:47 |
| jeblair | dhellmann: and maybe not worry too much about the slave contention issue for now. | 21:47 |
| jeblair | dhellmann: (but keep it in mind, and we see how it goes) | 21:47 |
| clarkb | mordred: looking. somehow today turned into a get nothign done day despite not being at the conference | 21:47 |
| dhellmann | jeblair: ok | 21:47 |
| clarkb | mordred: so that is just prep work for future potential awesome? | 21:48 |
| jeblair | clarkb: we're really productive here. :) | 21:48 |
| mordred | clarkb: yes. I need that... | 21:48 |
| jeblair | despite the person next to us who is yelling into facetime or google+ or whatever. | 21:48 |
| mordred | clarkb: for this: https://review.openstack.org/#/c/38184/ | 21:49 |
| clarkb | jeblair: I spent far to much time trying to figure out android SIP before deciding it wasn't important and I would just use a headset | 21:49 |
| hashar | mordred: I have mailed our Debian guru regarding having a package to not restart a service. We will see :) | 21:49 |
| mordred | clarkb: so that https://review.openstack.org/#/c/38176/ will make sense | 21:49 |
| hashar | I am off! have a good afternoon | 21:49 |
| mordred | clarkb: I suppose I could have marked them all witha topic... | 21:49 |
| clarkb | mordred: its fine | 21:49 |
| mordred | hashar: it's possible | 21:49 |
| jeblair | clarkb: it'd be nice to figure that out; but yeah, that's why we'll support pstn. :) | 21:49 |
| mordred | hashar: usually via use of a value in a defaults file | 21:49 |
| jeblair | hashar: thanks, good night! | 21:49 |
| clarkb | mordred: shouldn't you have added jeepyb to the image build list too? | 21:49 |
| mordred | clarkb: did I not? | 21:50 |
| clarkb | mordred: you only added it to the wrap script | 21:50 |
| clarkb | mordred: I left a -1 with ^ | 21:51 |
| mordred | clarkb: https://review.openstack.org/#/c/38180/ | 21:52 |
| hashar | mordred: I guess so :) | 21:52 |
| hashar | mordred: but I like asking Faidon dumb questions and this way you can follow up with him regarding packaging. I thought it was a good idea to get you two in touch | 21:52 |
| hashar | touch/contact | 21:52 |
| *** changbl has quit IRC | 21:53 | |
| clarkb | mordred: your dependency chains are too long and complicated .. :P | 21:54 |
| clarkb | mordred: you realize that by having "Use pbr" be the root of that list nothing is merging? we are all afraid of the pbr | 21:54 |
| *** johnthetubaguy has quit IRC | 21:54 | |
| *** hashar has left #openstack-infra | 21:56 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Add jeepyb to the list of cached repos https://review.openstack.org/36923 | 21:57 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Split projects list into its own file https://review.openstack.org/38181 | 21:57 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Add missing projects to pre-cache list https://review.openstack.org/38180 | 21:57 |
| mordred | clarkb: use pbr is not at the root of that list | 21:57 |
| mordred | I undid that a while ago :) | 21:57 |
| mordred | clarkb: well, I uploaded a new version of the patch with the thing fixed - I didn't catch your approval | 21:58 |
| clarkb | mordred: there are ~4 changes that sit atop pbr | 21:58 |
| clarkb | mordred: np, I will rereview and push through | 21:58 |
| mordred | clarkb: yes. the 4 atop pbr are actually part of the same work series | 21:58 |
| mordred | which is the "clean up build" | 21:59 |
| clarkb | mordred: any chance we can put pbr at the top of the list instead of the bottom? | 21:59 |
| *** hashar has joined #openstack-infra | 21:59 | |
| mordred | clarkb: hrm. perhaps? | 21:59 |
| clarkb | mordred: after the ensure latest setuptools and review.o.o not updating jeepyb and so on I really don't want to approve that change while you guys are OSCONing | 21:59 |
| clarkb | mordred: but the pyflakes and testr stuff are safe I think | 22:00 |
| mordred | :) | 22:00 |
| *** cody-somerville has quit IRC | 22:01 | |
| *** vipul is now known as vipul-away | 22:01 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Use pbr for build. https://review.openstack.org/33204 | 22:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Use testr and flake8. https://review.openstack.org/33205 | 22:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Fixed some line-length and whitespace warnings. https://review.openstack.org/33206 | 22:02 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Fixed pyflakes issues. https://review.openstack.org/33208 | 22:02 |
| mordred | clarkb: that patch bomb is for you | 22:02 |
| clarkb | danke | 22:02 |
| dhellmann | mordred: https://bitbucket.org/kang/python-keyring-lib/issue/105/pytest-runner-should-be-in-test | 22:04 |
| *** reed has joined #openstack-infra | 22:04 | |
| dhellmann | mordred: if you have time to weigh in on that, maybe you have another point to make our case | 22:05 |
| *** pcrews has quit IRC | 22:05 | |
| *** burt has quit IRC | 22:05 | |
| *** lbragstad has quit IRC | 22:06 | |
| *** UtahDave has joined #openstack-infra | 22:07 | |
| *** mestery has quit IRC | 22:07 | |
| clarkb | mordred: can you see my comment on 38181? | 22:08 |
| clarkb | mordred: the other two lgtm and I can approve them when check tests come back | 22:08 |
| * clarkb starts reviewing the other stack of changes | 22:08 | |
| *** vijendar has quit IRC | 22:10 | |
| *** vipul-away is now known as vipul | 22:12 | |
| *** hashar has quit IRC | 22:12 | |
| clarkb | mordred: and that second stack lgtm without pbr at the bottom of it | 22:15 |
| clarkb | jeblair: second opinion on my comments here https://review.openstack.org/#/c/38181/2/devstack-vm-gate-wrap.sh would be great | 22:16 |
| *** mestery has joined #openstack-infra | 22:16 | |
| clarkb | mordred: is https://review.openstack.org/#/c/35730/ one of the changes that went from the feature branch to master? | 22:17 |
| *** mrodden has quit IRC | 22:19 | |
| *** rfolco has quit IRC | 22:21 | |
| *** pcrews has joined #openstack-infra | 22:23 | |
| *** ladquin has quit IRC | 22:24 | |
| *** cody-somerville has joined #openstack-infra | 22:25 | |
| *** _TheDodd_ has quit IRC | 22:27 | |
| clarkb | fungi: https://review.openstack.org/#/c/38181/2/devstack-vm-gate-wrap.sh | 22:27 |
| clarkb | just trying to get a second opinion on my comment there as I can remove the -1 if I am just derping | 22:27 |
| *** zyluo has joined #openstack-infra | 22:28 | |
| *** atiwari has quit IRC | 22:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add gate job for CI docs https://review.openstack.org/36295 | 22:33 |
| *** jhesketh__ has joined #openstack-infra | 22:33 | |
| *** datsun180b has quit IRC | 22:36 | |
| *** zul has quit IRC | 22:38 | |
| *** sarob has joined #openstack-infra | 22:38 | |
| *** zul has joined #openstack-infra | 22:39 | |
| sdague | fungi: is it possible to spin up a log-dev environment so that we can test these filter changes on real stuff? | 22:39 |
| *** leifmadsen has quit IRC | 22:39 | |
| *** sarob_ has joined #openstack-infra | 22:39 | |
| sdague | btw, I have a new version on wsgi with content negotiation, will clean it up and get it out there shortly | 22:39 |
| *** mrodden has joined #openstack-infra | 22:40 | |
| clarkb | sdague: would a second vhost work? | 22:40 |
| *** pentameter has quit IRC | 22:40 | |
| *** dkliban has joined #openstack-infra | 22:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use ssl for PyPI uploads https://review.openstack.org/36914 | 22:41 |
| *** leifmadsen has joined #openstack-infra | 22:41 | |
| sdague | clarkb: yep | 22:41 |
| *** markmcclain has quit IRC | 22:42 | |
| clarkb | FYI if you see pypi uploads fail ^ is probably to blame. We can rollback and rerun the upload jobs if necessary but we don't expect it to be a problem | 22:42 |
| *** markmcclain has joined #openstack-infra | 22:42 | |
| *** sarob has quit IRC | 22:43 | |
| *** sdake_ has quit IRC | 22:43 | |
| dstufft | yay ssl | 22:44 |
| *** prad has quit IRC | 22:45 | |
| *** zyluo has quit IRC | 22:45 | |
| dstufft | Ng: ping | 22:46 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Pass venv name not py version to run-tox.sh https://review.openstack.org/38218 | 22:47 |
| *** krtaylor has joined #openstack-infra | 22:47 | |
| clarkb | sdague: have you seen https://jenkins.openstack.org/job/gate-tempest-devstack-vm-postgres-full/34238/console before? Unable to communicate with identity service | 22:48 |
| *** fbo is now known as fbo_away | 22:49 | |
| *** alexpilotti_ has joined #openstack-infra | 22:51 | |
| clarkb | hmm looks like docs draft remounted read only after a hung task timeout and some nasty looking kernel stacktraces | 22:51 |
| *** avtar has quit IRC | 22:52 | |
| *** vipul is now known as vipul-away | 22:52 | |
| *** ^d has joined #openstack-infra | 22:52 | |
| *** ^d has joined #openstack-infra | 22:52 | |
| *** mjfork_ has joined #openstack-infra | 22:52 | |
| *** vipul-away is now known as vipul | 22:53 | |
| *** zyluo_ has joined #openstack-infra | 22:54 | |
| *** dkliban has quit IRC | 22:55 | |
| *** rwsu has quit IRC | 22:56 | |
| dhellmann | clarkb: when you have a minute, could you take another look at https://review.openstack.org/#/c/36893/ for me? | 22:57 |
| dhellmann | clarkb: it's got 2 +2 but I had to rebase to get it to merge cleanly | 22:57 |
| sdague | clarkb: you got the logs link for that? | 22:57 |
| dhellmann | clarkb: no rush | 22:57 |
| clarkb | sdague: http://logs.openstack.org/23/36923/3/gate/gate-tempest-devstack-vm-postgres-full/34238/logs/ | 22:58 |
| dhellmann | what does "LOST" mean as a check job status? | 22:58 |
| clarkb | dhellmann: yeah currently trying to decide if Ishould attempt a ninja remount of the docs draft filesystem or fsck or reboot | 22:58 |
| dhellmann | ah, is that why I got a LOST? | 22:59 |
| dhellmann | it was a docs job | 22:59 |
| sdague | clarkb: keystone hit a port conflict | 22:59 |
| clarkb | dhellmann: if it was a docs job the nprobably | 22:59 |
| sdague | and didn't start | 22:59 |
| *** cherriges has joined #openstack-infra | 22:59 | |
| clarkb | sdague: nice | 22:59 |
| sdague | which is......weird | 22:59 |
| dhellmann | clarkb: ok, I'll stop pestering you then :-) | 22:59 |
| clarkb | #status notice Doc jobs currently failing due to filesystem errors. Infra is currently working on a fix. | 23:00 |
| clarkb | :/ we lost the bot | 23:00 |
| *** gyee has quit IRC | 23:00 | |
| *** odyi has quit IRC | 23:00 | |
| *** ^demon has quit IRC | 23:00 | |
| *** mjfork has quit IRC | 23:00 | |
| *** alexpilotti has quit IRC | 23:00 | |
| *** arosen1 has quit IRC | 23:00 | |
| *** alexpilotti_ is now known as alexpilotti | 23:00 | |
| *** cherriges is now known as odyi | 23:00 | |
| *** odyi has joined #openstack-infra | 23:00 | |
| zyluo_ | clarkb: thanks had the same question | 23:01 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add utility to htmlify screen logs for devstack runs https://review.openstack.org/37138 | 23:01 |
| *** arosen has joined #openstack-infra | 23:02 | |
| *** arosen has quit IRC | 23:02 | |
| *** arosen has joined #openstack-infra | 23:02 | |
| sdague | clarkb: got a chance to sanity check that? I'm sure there are a few things I borked up in it, but I'm hoping this is a more fruitful direction | 23:02 |
| *** gyee has joined #openstack-infra | 23:02 | |
| clarkb | sdague: when I sort out docs-draft I will switch back to code review | 23:02 |
| sdague | the performance on that on my laptop was pretty good, log results come back basically immediately | 23:02 |
| sdague | cool, no worries | 23:02 |
| *** arborism has joined #openstack-infra | 23:08 | |
| clarkb | jeblair: you about? | 23:10 |
| clarkb | jeblair: do you have opinions on correcting docs-draft? | 23:10 |
| clarkb | remount is failing. I think I need to unmount, fsck, then mount | 23:11 |
| clarkb | fungi: mordred ^ | 23:11 |
| jeblair | oh hi | 23:12 |
| jeblair | clarkb: dmesg says there were io errors | 23:13 |
| clarkb | jeblair: see dmesg on static.o.o. docs-draft was remounted read only after a timeout | 23:13 |
| clarkb | jeblair: yeah | 23:13 |
| jeblair | [5799369.774706] end_request: I/O error, dev xvdh, sector 210133272 | 23:13 |
| jeblair | clarkb: yeah, i think it's worth trying unmount/mount. | 23:14 |
| jeblair | clarkb: you want to do that? | 23:14 |
| clarkb | jeblair: sure. do you think I need to try a fsck in the middle? | 23:14 |
| jeblair | clarkb: couldn't hurt | 23:15 |
| jeblair | Alex_Gaynor: ping | 23:15 |
| Alex_Gaynor | jeblair: pong | 23:15 |
| clarkb | jeblair: are you in /srv/static/docs-draft? device is busy cannot unmount | 23:15 |
| jeblair | clarkb: done, sorry | 23:15 |
| *** vipul is now known as vipul-away | 23:16 | |
| clarkb | bah I was in there too | 23:16 |
| *** gyee has quit IRC | 23:16 | |
| *** tjones has joined #openstack-infra | 23:16 | |
| clarkb | looks like one other person has a root shell in there | 23:17 |
| clarkb | don't make me kill your root shell :P | 23:17 |
| tjones | is it possible to ask for a feature request on the bug dashboard? Would i file a feature request bug for something like that? | 23:17 |
| clarkb | brb | 23:17 |
| *** rahmu has quit IRC | 23:17 | |
| *** gaelL has quit IRC | 23:18 | |
| jeblair | Alex_Gaynor: we're seeing device timeouts with a cinder volume on a rax nova node (static.openstack.org) | 23:18 |
| *** vipul-away is now known as vipul | 23:18 | |
| Alex_Gaynor | jeblair: do you know what DC that node is in? | 23:19 |
| jeblair | Alex_Gaynor: is there useful feedback related to that we could provide to someone at rax? | 23:19 |
| *** jd__ has quit IRC | 23:19 | |
| jeblair | Alex_Gaynor: dfw | 23:19 |
| *** UtahDave has quit IRC | 23:19 | |
| Alex_Gaynor | jeblair: ok, nothing about about that on the status page (there is some sort of incident at ORD). Let me find out. | 23:19 |
| *** saper has quit IRC | 23:21 | |
| Alex_Gaynor | jeblair: is that what all teh failing doc builds are about? | 23:21 |
| clarkb | jeblair: I think the last shell that is causing problems is part of one of your screens | 23:22 |
| clarkb | jeblair: any chance you could attach to that and cd? | 23:22 |
| jeblair | Alex_Gaynor: yeah | 23:22 |
| jeblair | clarkb: oh, sorry, i'm sure that's done i'll close it | 23:22 |
| Alex_Gaynor | jeblair: I'm trying to find out what the procedure for something like this is, in the meantime your best bet is probably to open a supporrt ticket | 23:23 |
| jeblair | clarkb: done | 23:23 |
| clarkb | jeblair: `fsck /dev/mapper/main-docs--draft` ? any special options I should use? | 23:24 |
| jeblair | clarkb: i don't think so | 23:25 |
| *** pcrews has quit IRC | 23:25 | |
| *** jd__ has joined #openstack-infra | 23:25 | |
| *** gaelL has joined #openstack-infra | 23:25 | |
| clarkb | it is running. "docs-draft contains a file system with errors, check forced." | 23:25 |
| *** pcrews has joined #openstack-infra | 23:27 | |
| *** rahmu has joined #openstack-infra | 23:27 | |
| *** saper has joined #openstack-infra | 23:27 | |
| clarkb | jeblair lost+found not found. did you want to create one? I thought you ran into this before | 23:29 |
| jeblair | clarkb: the change to not delete that is in review? | 23:29 |
| jeblair | clarkb: not sure it's important; if you or it create it, it will probably be deleted | 23:29 |
| jeblair | clarkb: whatever you need to do for the moment. i don't think we care about lost data though. | 23:30 |
| clarkb | I see. Our finds are deleting it? I will create it then make sure there is something in review later | 23:30 |
| jeblair | clarkb: there is something in review already | 23:30 |
| clarkb | ok, will review then | 23:30 |
| clarkb | I am telling fsck to fix the other problems it is seeing | 23:30 |
| jeblair | https://review.openstack.org/#/c/37805/ | 23:30 |
| jeblair | k | 23:30 |
| clarkb | ok fsck done. Will mount in a sec | 23:31 |
| clarkb | it is mounted | 23:31 |
| *** dhellmann is now known as dhellmann_ | 23:31 | |
| clarkb | https://jenkins.openstack.org/job/gate-nova-docs/28064/ should not go UNSTABLE | 23:32 |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/config: Create new repo to host legacy heat-cfn client. https://review.openstack.org/38226 | 23:32 |
| jeblair | i think there may be more transient network errors recently... | 23:33 |
| *** zyluo_ has quit IRC | 23:33 | |
| clarkb | jeblair: yeah | 23:33 |
| *** zyluo has joined #openstack-infra | 23:33 | |
| jeblair | there's the unstable jobs from earlier (which were scp problems)... | 23:33 |
| clarkb | jeblair: fungi I have reviewed 37805. I think the change is missing some \'s to escape newlines | 23:34 |
| jeblair | but i also just saw something that failed to download a tarball from tarballs.o.o | 23:34 |
| jeblair | which is not on static, so is a different host | 23:34 |
| jeblair | though it was a grenade job, so i suppose it could have been something other than rax (hp or internet); though since we saw scp failures on static slaves i suspect it's localized to rax | 23:35 |
| clarkb | sdague: does that keystone failure during devstack have an existing bug? | 23:36 |
| clarkb | the port conflict thing? | 23:36 |
| sdague | clarkb: not that I know of | 23:36 |
| sdague | that's the first time I've ever seen it | 23:36 |
| clarkb | sdague: gut feeling is that a keystone or devstack bug? | 23:36 |
| clarkb | oh, I wonder if we ran two tests on that host | 23:36 |
| clarkb | the host is no longer in jenkins so I will need to wait for logstash to catch up to do a search | 23:37 |
| jog0 | do we use trunk pbr or releases of it? | 23:37 |
| clarkb | btw the addition of all these additional tempest jobs really slows logstash down :) | 23:38 |
| jeblair | ooh, which host? | 23:38 |
| clarkb | jeblair: http://logs.openstack.org/23/36923/3/gate/gate-tempest-devstack-vm-postgres-full/34238/console.html | 23:38 |
| clarkb | jeblair: if you look at the keystoen log for that job you see what caused the failure | 23:38 |
| clarkb | jeblair: I think typically it is releases but devstack-gate driven tests will use trunk | 23:39 |
| clarkb | er jog0 ^ | 23:39 |
| jeblair | i suspect a bug causing multiple usages of devstack hosts, we now have logs that should help me track that down | 23:39 |
| jog0 | clarkb: thaks is that true for stock devstack as well? | 23:39 |
| clarkb | jeblair: ok I will start with an openstack-ci bug and recheck against that | 23:39 |
| jog0 | as I am eager to try https://review.openstack.org/#/c/38000/ | 23:40 |
| *** tjones has quit IRC | 23:40 | |
| clarkb | jog0: it is not true for stock devstack iirc. devstack-gate does a pre step thing where it updates all repos, checks out the ref(s) being testing and gets them all ready | 23:40 |
| clarkb | then devstack runs | 23:40 |
| jeblair | clarkb: that doesn't look like what i was tracking down | 23:40 |
| jeblair | clarkb: why do you think it was used twice? | 23:40 |
| *** michchap has joined #openstack-infra | 23:40 | |
| jog0 | so clarkb, that means mordred needs to cut a new pbr? | 23:41 |
| clarkb | jeblair: because of the port conflict. I am guessing that there was an existing keystone process running on that host from a previous run | 23:41 |
| *** bclifford has left #openstack-infra | 23:41 | |
| *** michchap has quit IRC | 23:41 | |
| clarkb | jog0: if you want to test it easily yes | 23:41 |
| clarkb | jog0: otherwise you could install it from 38000 globally then devstack projects should use that version | 23:41 |
| *** michchap has joined #openstack-infra | 23:41 | |
| jeblair | clarkb: yes it was used twice | 23:43 |
| clarkb | jeblair: thank you. I am submitting a new bug. Would you prefer I update an old one? | 23:43 |
| jeblair | http://paste.openstack.org/show/41313/ | 23:43 |
| jeblair | clarkb: new bug; assign to me | 23:43 |
| clarkb | jeblair: will do | 23:43 |
| *** bclifford1 has joined #openstack-infra | 23:44 | |
| jog0 | clarkb: I tried doing that before it merged and must have missed a step to get projects to use it, I just ran python setup.py develop in pbr and nova | 23:45 |
| jog0 | but that wasn't enough | 23:45 |
| *** bclifford2 has joined #openstack-infra | 23:45 | |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1203905 | 23:45 |
| uvirtbot | Launchpad bug 1203905 in openstack-ci "Keystone fails to start due to a port conflict during gate-tempest test." [High,Triaged] | 23:45 |
| jeblair | clarkb, zaro: i think the log i pasted confirms that there's a gearman-plugin bug | 23:46 |
| *** tjones has joined #openstack-infra | 23:47 | |
| *** bclifford1 has quit IRC | 23:48 | |
| jeblair | clarkb, zaro: it already had 'devstack-used' as a label when the inprogress job ran the second time. | 23:48 |
| *** cody-somerville has quit IRC | 23:49 | |
| *** bclifford2 has quit IRC | 23:49 | |
| clarkb | jeblair: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#L800 is that not doing it? | 23:50 |
| *** zyluo has quit IRC | 23:51 | |
| clarkb | jog0: I would expect that to be sufficient unless the resulting version is outside of the bounds of what the individual projects allow | 23:51 |
| jeblair | clarkb: i believe the node label _is_ being correctly set | 23:52 |
| clarkb | jeblair: oh, but gearman is still scheduling things on that node | 23:52 |
| jeblair | clarkb: but jenkins and/or gearman-plugin are running the job anyway | 23:52 |
| *** alexpilotti has quit IRC | 23:53 | |
| jog0 | clarkb: hmm no pip seeding needed? | 23:53 |
| clarkb | jeblair: I think the inprogress job runs non concurrently. Is it possible there is a race between two jobs being scheduled on the same host waiting for the inprogress to change the label? | 23:53 |
| clarkb | jog0: I don't think so | 23:53 |
| clarkb | jog0: because you will end up with a newer version than is available from pypi installed so pip will prefer to use that version. The caveat to that is the new version has to be within the allowed bounds | 23:54 |
| jog0 | clarkb: trying out now | 23:54 |
| clarkb | jeblair: having a single executor per slaves should avoid that though | 23:56 |
| *** ^d has quit IRC | 23:57 | |
| *** rnirmal has quit IRC | 23:58 | |
| jeblair | clarkb: i think devstack jobs wait on the inprogress job to complete; if i'm wrong about that then that could definitely be a problem. | 23:58 |
| mordred | jog0: ping | 23:58 |
| clarkb | jeblair: comparing timestamps I am not sure it is waiting | 23:59 |
| mordred | jog0: we should be using trunk pbr in devstack | 23:59 |
| jeblair | clarkb: oops | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!