| *** ajmiller has quit IRC | 00:00 | |
| fungi | nibalizer: it's utc midnight now, so free to start running that script at any time. people are now past the deadline for adjusting their preferred e-mail addresses in gerrit | 00:00 |
|---|---|---|
| *** akshai has joined #openstack-infra | 00:00 | |
| *** tzn has joined #openstack-infra | 00:01 | |
| *** lucas-afk has quit IRC | 00:02 | |
| *** rhallisey has quit IRC | 00:03 | |
| *** tzn has quit IRC | 00:03 | |
| *** achanda has joined #openstack-infra | 00:03 | |
| pabelanger | okay, so debian-jessie dib was successful with uploads happening now | 00:04 |
| pabelanger | we should be able to check rax shortly | 00:04 |
| dhellmann | is etherpad ok? I can't get a couple of pads to load | 00:04 |
| clarkb | dhellmann: db backups happen at 0000UTC | 00:04 |
| dhellmann | clarkb : ok, thanks | 00:05 |
| fungi | pabelanger: in that case mind copying the qcow2 into /var/log/nodepool/images/ and letting us know the download url so zigo can grab a copy to mount locally and inspect? | 00:05 |
| clarkb | pabelanger: rax is the longest uploads, osic and bluebox have the image uploaded though | 00:05 |
| pabelanger | fungi: sure, 1 min | 00:05 |
| pabelanger | clarkb: ya, we are only launching on rax and ovh currently | 00:06 |
| fungi | pabelanger: no rush, i just know he's eager to give it a once over and see if there's more needed on the dib side of things | 00:06 |
| *** darrenc_ has joined #openstack-infra | 00:06 | |
| *** fverbs has joined #openstack-infra | 00:06 | |
| *** rm_work has quit IRC | 00:06 | |
| *** mjb has quit IRC | 00:06 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:06 | |
| *** darrenc has quit IRC | 00:07 | |
| *** mcarden has quit IRC | 00:07 | |
| *** flwang has quit IRC | 00:07 | |
| *** gus has quit IRC | 00:07 | |
| *** vkramskikh has quit IRC | 00:07 | |
| *** mcarden has joined #openstack-infra | 00:07 | |
| *** Jeffrey4l has quit IRC | 00:08 | |
| *** rm_work has joined #openstack-infra | 00:08 | |
| clarkb | pabelanger: oh, why? | 00:08 |
| *** lucasagomes has joined #openstack-infra | 00:08 | |
| *** mjb has joined #openstack-infra | 00:08 | |
| *** gus has joined #openstack-infra | 00:08 | |
| *** dims has joined #openstack-infra | 00:08 | |
| pabelanger | clarkb: not sure. I believe that was the only clouds originally tested | 00:09 |
| pabelanger | clarkb: so, I figure if rax comes only, we can enable the rest of them | 00:09 |
| *** flwang1 has joined #openstack-infra | 00:09 | |
| *** dims_ has quit IRC | 00:09 | |
| pabelanger | fungi: zigo: http://nodepool.openstack.org/176625.qcow2 is the debian-jessie dib | 00:09 |
| *** dims has quit IRC | 00:10 | |
| clarkb | ok one is booting in ovh now | 00:10 |
| pabelanger | ya | 00:11 |
| pabelanger | working on updating puppet-apply to get debian experimental | 00:11 |
| *** vkramskikh has joined #openstack-infra | 00:13 | |
| *** mtanino has quit IRC | 00:13 | |
| clarkb | and its ready | 00:13 |
| pabelanger | yup | 00:13 |
| pabelanger | nice | 00:13 |
| pabelanger | will have patch in 2mins | 00:14 |
| *** hichihara has joined #openstack-infra | 00:14 | |
| *** achanda has quit IRC | 00:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Run puppet-apply on debian jessie jenkins slaves https://review.openstack.org/293766 | 00:15 |
| *** jokke_ has quit IRC | 00:16 | |
| *** sridhar_ram has quit IRC | 00:16 | |
| *** jokke_ has joined #openstack-infra | 00:16 | |
| clarkb | I logged in disk looks good | 00:16 |
| pabelanger | nice | 00:17 |
| *** yamahata has joined #openstack-infra | 00:18 | |
| clarkb | pabelanger: that change needs a corresponding one to run the apply test on jessie | 00:19 |
| clarkb | which might need to go atop my change or under it, I don't care which order but reabsing likely necessary | 00:20 |
| pabelanger | clarkb: Yup, just running tox now on project-config | 00:20 |
| openstackgerrit | Merged openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes https://review.openstack.org/270273 | 00:20 |
| *** timothyb89 has quit IRC | 00:20 | |
| *** fawadkhaliq has quit IRC | 00:20 | |
| *** fawadkhaliq has joined #openstack-infra | 00:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental puppet-apply job for debian-jessie https://review.openstack.org/293767 | 00:23 |
| pabelanger | and experimental job | 00:23 |
| pabelanger | we can likely add bindep-fallback too | 00:23 |
| pabelanger | but need a drink first | 00:23 |
| *** flwang1 has quit IRC | 00:23 | |
| *** timothyb89 has joined #openstack-infra | 00:25 | |
| *** darrenc_ is now known as darrenc | 00:25 | |
| *** flepied has joined #openstack-infra | 00:26 | |
| *** flwang1 has joined #openstack-infra | 00:26 | |
| *** baoli has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Run puppet-apply on debian jessie jenkins slaves https://review.openstack.org/293766 | 00:28 |
| *** baoli has quit IRC | 00:30 | |
| tonyb | nibalizer: Feel free to generate the rolls anytime from now (for the next 23.5 hours) | 00:30 |
| *** Swami has quit IRC | 00:35 | |
| *** adrian_otto has quit IRC | 00:35 | |
| *** asettle has joined #openstack-infra | 00:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental debian-jessie job for bindep-fallback https://review.openstack.org/293769 | 00:35 |
| *** zz_dimtruck is now known as dimtruck | 00:36 | |
| *** thorst has joined #openstack-infra | 00:37 | |
| *** salv-orl_ has quit IRC | 00:40 | |
| fungi | nibalizer: also, as i said, i'm happy to walk you through it, and run it myself again too so we can compare results if you want | 00:40 |
| fungi | i've been running it... a lot | 00:40 |
| *** yamahata has quit IRC | 00:41 | |
| fungi | i just want to be certain that, as a ptl candidate in a contested race, i'm nowhere in the chain of custody of the electorate data between gerrit and the officials | 00:41 |
| mordred | heya | 00:41 |
| mordred | I miss anything good? | 00:42 |
| clarkb | we are almost off of devstack-trusty | 00:42 |
| clarkb | we have debian slaves now | 00:42 |
| mordred | woot | 00:42 |
| mordred | ! | 00:42 |
| mordred | everyone is awesome | 00:42 |
| fungi | cats and dogs living together | 00:43 |
| fungi | mass hysteria | 00:43 |
| clarkb | jackfruit beer is surprisingly good | 00:43 |
| clarkb | thtas about all I have | 00:43 |
| fungi | mordred: we're going to have an actual infra ptl election for, like, the first time ever | 00:43 |
| *** cloudtrainme has joined #openstack-infra | 00:45 | |
| mordred | fungi: I'm excited! | 00:47 |
| mordred | fungi: we're like a real thing | 00:47 |
| fungi | all projecty and stuff | 00:48 |
| mordred | fungi: does this mean we have to start using rabbit for more things? especially for things it wasn't designed to be used for? | 00:48 |
| fungi | mordred: we're going to use it as our distributed lock manager. that's already been decided | 00:49 |
| mordred | fungi: AWESOME | 00:49 |
| fungi | or maybe as a syslog replacement | 00:49 |
| fungi | ooh, rabbit as an init system! | 00:49 |
| fungi | take that, lennart! | 00:49 |
| *** fawadkhaliq has quit IRC | 00:51 | |
| *** fawadkhaliq has joined #openstack-infra | 00:51 | |
| bkero | I'm sure he'll be devastated. | 00:51 |
| *** dims has joined #openstack-infra | 00:52 | |
| ianw | does anyone know how to get iced-tea to stop warning me about the terrible things the rax java console will do to my system | 00:52 |
| clarkb | mix it iwth lemonade | 00:53 |
| ianw | it makes it almost impossible to get the early boot output, since rax likes to disconnect the damn thing every time you reboot | 00:53 |
| fungi | yes, click the okay button (every single time) | 00:53 |
| *** sputnik13 has quit IRC | 00:53 | |
| fungi | and yeah, definitely a pain that you get disconnected when the instance drops its vga | 00:53 |
| fungi | i wonder if nova console works there? or does xen not support that? | 00:54 |
| *** chlong has joined #openstack-infra | 00:55 | |
| *** asettle has quit IRC | 00:56 | |
| *** baoli has joined #openstack-infra | 00:56 | |
| bkero | ianw: 2>/dev/null? :) | 00:57 |
| *** fverbs has quit IRC | 00:58 | |
| ianw | http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=2189 <- so yeah, i'm not crazy, it is a bug | 00:59 |
| openstack | icedtea.classpath.org bug 2189 in Plugin "Unable to remove security pop up" [Enhancement,Resolved: fixed] - Assigned to jvanek | 00:59 |
| *** baoli has quit IRC | 00:59 | |
| *** akshai_ has joined #openstack-infra | 01:01 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 01:02 |
| *** ams_ has quit IRC | 01:04 | |
| *** kzaitsev_mb has quit IRC | 01:04 | |
| *** odyssey4me has quit IRC | 01:04 | |
| *** bpokorny has quit IRC | 01:04 | |
| *** baoli has joined #openstack-infra | 01:05 | |
| *** akshai has quit IRC | 01:05 | |
| *** ams_ has joined #openstack-infra | 01:06 | |
| *** odyssey4me has joined #openstack-infra | 01:06 | |
| *** baoli has quit IRC | 01:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Cleanup of nodepool builder logging https://review.openstack.org/275982 | 01:07 |
| *** smarcet has quit IRC | 01:07 | |
| *** ramishra has quit IRC | 01:07 | |
| *** zhurong has joined #openstack-infra | 01:07 | |
| *** ramishra has joined #openstack-infra | 01:08 | |
| *** cloudtrainme has quit IRC | 01:11 | |
| *** jpr has joined #openstack-infra | 01:13 | |
| *** tphummel has quit IRC | 01:15 | |
| *** chlong has quit IRC | 01:18 | |
| *** asettle has joined #openstack-infra | 01:20 | |
| *** sflanigan has quit IRC | 01:20 | |
| *** IlyaG has quit IRC | 01:20 | |
| *** sdake has quit IRC | 01:22 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Refactor the v3 only job https://review.openstack.org/271128 | 01:22 |
| *** piet has quit IRC | 01:22 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the v3 only job in neutron https://review.openstack.org/293200 | 01:23 |
| *** baoli has joined #openstack-infra | 01:23 | |
| *** hrybacki has quit IRC | 01:25 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos https://review.openstack.org/293775 | 01:26 |
| *** achanda has joined #openstack-infra | 01:26 | |
| *** hrybacki has joined #openstack-infra | 01:26 | |
| *** apoorvad has quit IRC | 01:27 | |
| *** piet has joined #openstack-infra | 01:28 | |
| *** Daisy has joined #openstack-infra | 01:30 | |
| *** chlong has joined #openstack-infra | 01:31 | |
| *** Daisy has quit IRC | 01:32 | |
| *** Daisy has joined #openstack-infra | 01:33 | |
| *** jpr1 has joined #openstack-infra | 01:36 | |
| *** jpr2 has joined #openstack-infra | 01:37 | |
| *** jpr has quit IRC | 01:37 | |
| *** Daisy has quit IRC | 01:37 | |
| *** eliqiao_ has joined #openstack-infra | 01:40 | |
| *** jpr1 has quit IRC | 01:40 | |
| *** alex_xu has quit IRC | 01:42 | |
| *** eliqiao has quit IRC | 01:43 | |
| *** baoli has quit IRC | 01:43 | |
| *** alex_xu has joined #openstack-infra | 01:44 | |
| *** tqtran-afk has quit IRC | 01:48 | |
| *** piet has quit IRC | 01:49 | |
| *** alex_xu has quit IRC | 01:51 | |
| *** fawadkhaliq has quit IRC | 01:52 | |
| *** alex_xu has joined #openstack-infra | 01:53 | |
| *** fawadkhaliq has joined #openstack-infra | 01:53 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code https://review.openstack.org/289255 | 01:54 |
| *** woodster_ has quit IRC | 01:57 | |
| *** fawadkhaliq has quit IRC | 01:58 | |
| *** fawadkhaliq has joined #openstack-infra | 01:58 | |
| *** Daisy has joined #openstack-infra | 01:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:00 | |
| *** HeOS has quit IRC | 02:01 | |
| *** Daisy has quit IRC | 02:01 | |
| *** [HeOS] has joined #openstack-infra | 02:01 | |
| *** asselin_ has joined #openstack-infra | 02:01 | |
| *** Daisy has joined #openstack-infra | 02:02 | |
| *** bryan_att has quit IRC | 02:02 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos https://review.openstack.org/293775 | 02:02 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code https://review.openstack.org/289255 | 02:02 |
| *** salv-orlando has joined #openstack-infra | 02:03 | |
| *** asselin__ has joined #openstack-infra | 02:04 | |
| *** asselin_ has quit IRC | 02:06 | |
| Daisy | AJaeger: there ? | 02:08 |
| *** sflanigan has joined #openstack-infra | 02:08 | |
| *** ramishra has quit IRC | 02:09 | |
| *** thorst has quit IRC | 02:09 | |
| *** rabi has joined #openstack-infra | 02:09 | |
| *** rabi has quit IRC | 02:09 | |
| *** gildub has quit IRC | 02:09 | |
| *** thorst has joined #openstack-infra | 02:10 | |
| *** bmjen has quit IRC | 02:10 | |
| ianw | is there anyone with their key in the built images who could log into 166.78.135.183 and reset the root password for me? i don't want to reboot it, because i want to see what glean has done on its first boot | 02:11 |
| *** Hunner has quit IRC | 02:12 | |
| *** salv-orlando has quit IRC | 02:12 | |
| mordred | ianw: you want root password or your key added to it? | 02:13 |
| ianw | mordred: if you just set the root password i can log on in the console | 02:13 |
| mordred | ah | 02:14 |
| *** eliqiao_ is now known as eliqiao | 02:14 | |
| ianw | it's my vm, but i can't log into it :) | 02:14 |
| mordred | ianw: password asdf | 02:14 |
| *** aeng has quit IRC | 02:14 | |
| mordred | ianw: of course, now EVERYONE knows your root password ... :) | 02:14 |
| * jroll logs in and reboots | 02:14 | |
| *** aeng has joined #openstack-infra | 02:15 | |
| mordred | ianw: for nexttime - there is an additional element you can add, devuser, that will add a user to the image with the ssh public key of the person building the image | 02:15 |
| mordred | ianw: you might find that useful if your'e debugging dib images and stuff | 02:15 |
| *** fawadkhaliq has quit IRC | 02:15 | |
| ianw | mordred: yeah, i know, but this is the official build image pabelanger shared with me | 02:15 |
| mordred | ah. gotcha | 02:15 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos https://review.openstack.org/293775 | 02:15 |
| *** fawadkhaliq has joined #openstack-infra | 02:16 | |
| mordred | clarkb: I feel like you told me about mirror related patches yesterday and then I went drinking or something ... do you need me to look at osmething? | 02:16 |
| clarkb | mordred yes one sec | 02:16 |
| *** aeng has quit IRC | 02:16 | |
| clarkb | mordred https://review.openstack.org/#/c/293155/ and its parent | 02:17 |
| *** oanson has quit IRC | 02:17 | |
| *** oanson has joined #openstack-infra | 02:18 | |
| *** thorst has quit IRC | 02:18 | |
| *** ramishra has joined #openstack-infra | 02:18 | |
| ianw | hmm, maybe we have a race condition between network.service and glean | 02:19 |
| *** odyssey4me has quit IRC | 02:19 | |
| *** lucasagomes has quit IRC | 02:19 | |
| *** darrenc has quit IRC | 02:19 | |
| *** mriedem1 has quit IRC | 02:19 | |
| *** mattoliverau has quit IRC | 02:19 | |
| *** aspiers has quit IRC | 02:19 | |
| *** treaki__ has quit IRC | 02:19 | |
| *** binbincong has quit IRC | 02:19 | |
| *** yolanda has quit IRC | 02:19 | |
| *** sileht has quit IRC | 02:19 | |
| *** jraim has quit IRC | 02:19 | |
| *** vmbrasseur has quit IRC | 02:19 | |
| *** mtreinish has quit IRC | 02:19 | |
| *** tsufiev has quit IRC | 02:19 | |
| *** Douhet has quit IRC | 02:19 | |
| *** fkautz has quit IRC | 02:19 | |
| *** Ng has quit IRC | 02:19 | |
| *** zxiiro has quit IRC | 02:19 | |
| *** fdegir has quit IRC | 02:19 | |
| *** simonmcc has quit IRC | 02:19 | |
| *** ashtokolov has quit IRC | 02:19 | |
| *** dansmith has quit IRC | 02:19 | |
| *** yonglihe has quit IRC | 02:19 | |
| *** evgenyl has quit IRC | 02:19 | |
| *** mwhahaha has quit IRC | 02:19 | |
| *** Volundr has quit IRC | 02:19 | |
| *** mescanef has quit IRC | 02:19 | |
| *** cfarquhar has quit IRC | 02:19 | |
| mordred | ianw: yay! | 02:20 |
| *** dims has quit IRC | 02:20 | |
| *** treaki__ has joined #openstack-infra | 02:20 | |
| *** MarkAtwood has joined #openstack-infra | 02:21 | |
| *** IlyaG has joined #openstack-infra | 02:21 | |
| *** zhurong has quit IRC | 02:22 | |
| *** Daisy_ has joined #openstack-infra | 02:22 | |
| *** Daisy has quit IRC | 02:22 | |
| *** akshai_ has quit IRC | 02:22 | |
| *** Qiming has joined #openstack-infra | 02:23 | |
| Daisy_ | Hi, I think something wrong with OpenStack ID server. I cannot login to translate website. defstack website is not able to log in neither, an error code 500 returned. | 02:24 |
| *** lucasagomes has joined #openstack-infra | 02:24 | |
| *** odyssey4me has joined #openstack-infra | 02:25 | |
| *** darrenc has joined #openstack-infra | 02:25 | |
| *** mriedem1 has joined #openstack-infra | 02:25 | |
| *** mattoliverau has joined #openstack-infra | 02:25 | |
| *** aspiers has joined #openstack-infra | 02:25 | |
| *** binbincong has joined #openstack-infra | 02:25 | |
| *** yolanda has joined #openstack-infra | 02:25 | |
| *** sileht has joined #openstack-infra | 02:25 | |
| *** jraim has joined #openstack-infra | 02:25 | |
| *** vmbrasseur has joined #openstack-infra | 02:25 | |
| *** mtreinish has joined #openstack-infra | 02:25 | |
| *** tsufiev has joined #openstack-infra | 02:25 | |
| *** Douhet has joined #openstack-infra | 02:25 | |
| *** fkautz has joined #openstack-infra | 02:25 | |
| *** Ng has joined #openstack-infra | 02:25 | |
| *** zxiiro has joined #openstack-infra | 02:25 | |
| *** fdegir has joined #openstack-infra | 02:25 | |
| *** simonmcc has joined #openstack-infra | 02:25 | |
| *** ashtokolov has joined #openstack-infra | 02:25 | |
| *** dansmith has joined #openstack-infra | 02:25 | |
| *** yonglihe has joined #openstack-infra | 02:25 | |
| *** evgenyl has joined #openstack-infra | 02:25 | |
| *** mwhahaha has joined #openstack-infra | 02:25 | |
| *** Volundr has joined #openstack-infra | 02:25 | |
| *** mescanef has joined #openstack-infra | 02:25 | |
| *** cfarquhar has joined #openstack-infra | 02:25 | |
| *** aviau has quit IRC | 02:25 | |
| clarkb | Daisy_: it has been having trouble today, smarcet was working on it | 02:25 |
| Daisy_ | Thanks for the information. | 02:26 |
| *** aviau has joined #openstack-infra | 02:26 | |
| *** fawadkhaliq has quit IRC | 02:26 | |
| Daisy_ | have to wait then. | 02:26 |
| *** aviau has quit IRC | 02:26 | |
| *** fawadkhaliq has joined #openstack-infra | 02:26 | |
| *** IlyaG has quit IRC | 02:26 | |
| *** aviau has joined #openstack-infra | 02:26 | |
| ianw | mordred: db1afd64 (Monty Taylor 2015-06-27 12:49:23 -0400 3) After=network.service network.target | 02:26 |
| ianw | maybe that's what is intended | 02:26 |
| *** madhu_ak has quit IRC | 02:26 | |
| *** johnsom_ has joined #openstack-infra | 02:28 | |
| *** johnsom has quit IRC | 02:28 | |
| *** kzaitsev_mb has quit IRC | 02:28 | |
| *** zhurong has joined #openstack-infra | 02:29 | |
| mordred | clarkb: landed parent patch. child patch failing tests - also put a -1 on it with a couple of comments | 02:29 |
| mordred | clarkb: if we're going to have this parameterized, do we need automatoin aroud creating afs volumes? | 02:30 |
| mordred | ianw: well ... assume my systemd is shaky at best | 02:30 |
| clarkb | mordred: no idea | 02:31 |
| clarkb | I guess we arent doing large aggregate volume because that isnt how afs works | 02:31 |
| *** tphummel has joined #openstack-infra | 02:32 | |
| mordred | right. we have volume-per-mirror so that we can release each one when we update it | 02:32 |
| *** akshai has joined #openstack-infra | 02:32 | |
| openstackgerrit | Merged openstack-infra/system-config: Make reprepro useable on multiple mirrors https://review.openstack.org/293154 | 02:36 |
| *** Daisy_ has quit IRC | 02:36 | |
| *** Daisy has joined #openstack-infra | 02:36 | |
| *** fawadkhaliq has quit IRC | 02:38 | |
| *** fawadkhaliq has joined #openstack-infra | 02:38 | |
| *** piet has joined #openstack-infra | 02:40 | |
| *** fawadkhaliq has quit IRC | 02:43 | |
| ianw | mordred: this would explain why it comes up much faster the second boot -- at that point glean has overwritten the network scripts. the first time, it waits for network.service which tries to dhcp configure the interfaces. did you possibly hit something about doing ifup before "network.service" was up? | 02:44 |
| *** ianychoi3 is now known as ianychoi | 02:45 | |
| mordred | ianw: possibly? I know there are screwy things related to having interfaces configured to listen for dhcp in the image as it relates to udev and stuff | 02:45 |
| mordred | I want to say you have to have _something_ configured for the interfaces or something in udev doesn't fire properly - but if you have dhcp, then the udev rule will ifup it before glean can rewrite it | 02:46 |
| mordred | but it works a little different on centos and ubuntu | 02:46 |
| mordred | ianw: consider that I'm perhaps just stupid | 02:47 |
| *** achanda has quit IRC | 02:48 | |
| *** achanda has joined #openstack-infra | 02:50 | |
| *** jpr2 has quit IRC | 02:50 | |
| ianw | well, just me booting the offiical image it took the network about 2 minutes to come up, but it did come up. i don't know if that's long enough to timeout the node getting registered | 02:50 |
| *** Daisy_ has joined #openstack-infra | 02:51 | |
| *** Daisy has quit IRC | 02:51 | |
| SamYaple | dont you people ever sleep? | 02:52 |
| *** achanda has quit IRC | 02:52 | |
| greghaynes | we sleep when waiting for test results | 02:53 |
| tonyb | nano-sleeps | 02:53 |
| SamYaple | "its compiling" | 02:54 |
| SamYaple | i see no reason to stop using that excuse | 02:54 |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** achanda has joined #openstack-infra | 02:55 | |
| *** asettle has quit IRC | 02:56 | |
| *** Qiming has quit IRC | 02:57 | |
| *** Qiming has joined #openstack-infra | 02:57 | |
| *** asettle has joined #openstack-infra | 02:59 | |
| EmilienM | clarkb: ty for https://review.openstack.org/#/c/293719/ | 03:00 |
| *** aspiers has quit IRC | 03:02 | |
| EmilienM | ianw: can you look at https://review.openstack.org/#/c/293749/ please? | 03:03 |
| *** asettle has quit IRC | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** armax has joined #openstack-infra | 03:09 | |
| *** piet has quit IRC | 03:10 | |
| prometheanfire | greghaynes: https://review.openstack.org/290894 again? | 03:12 |
| prometheanfire | need it for the project-config stuff | 03:12 |
| *** dimtruck is now known as zz_dimtruck | 03:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Add puppet-pacemaker to tripleo project list https://review.openstack.org/293749 | 03:14 |
| EmilienM | ianw: thx | 03:14 |
| *** markvoelker has quit IRC | 03:15 | |
| *** thorst has joined #openstack-infra | 03:16 | |
| *** salv-orlando has joined #openstack-infra | 03:17 | |
| *** yamahata has joined #openstack-infra | 03:22 | |
| *** thorst has quit IRC | 03:23 | |
| *** verdurin has quit IRC | 03:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:25 | |
| *** salv-orlando has quit IRC | 03:26 | |
| mordred | clarkb, SamYaple: well I'll be damned - task import works on vexxhost | 03:29 |
| *** claudiub|2 has quit IRC | 03:29 | |
| *** verdurin has joined #openstack-infra | 03:31 | |
| *** kzaitsev_mb has quit IRC | 03:31 | |
| mordred | clarkb: AND it converted the format AND it boots ... and it looks like vexxhost supports qcow2 and raw formats | 03:33 |
| mordred | once again, mnaser's cloud is awesome | 03:33 |
| clarkb | ya we direct upload qcow2 so its fine there | 03:34 |
| mordred | yah | 03:34 |
| mordred | just saying - it does seem to be a working function in stock glance | 03:34 |
| clarkb | ovh and rax are the ones | 03:34 |
| mordred | clarkb: I will now try ovh | 03:34 |
| clarkb | and I bet rax wont do qcow2 to vhd | 03:34 |
| mordred | clarkb: it seems to have worked at ovh | 03:35 |
| mordred | testing booting from it | 03:36 |
| clarkb | it boggles the mind why these are tasks | 03:36 |
| mordred | I know | 03:36 |
| mordred | and also undocumented | 03:36 |
| mordred | however | 03:36 |
| mordred | it seems that if we just put our images on a web server, we could only publish qcow2 images and reduce our size cost by at least a raw | 03:37 |
| *** IlyaG has joined #openstack-infra | 03:37 | |
| mordred | SamYaple: you win the gold star for teaching me even more crazy things about glance | 03:38 |
| *** verdurin has quit IRC | 03:39 | |
| clarkb | I mean why is each task not its own api verb | 03:39 |
| *** gildub has joined #openstack-infra | 03:40 | |
| mordred | clarkb: yah. I know | 03:40 |
| *** mriedem1 has quit IRC | 03:40 | |
| mordred | clarkb: I have booted a server in ovh from the image that I imported with the import task that convered form qcow2 to raw | 03:40 |
| clarkb | neat | 03:41 |
| *** jamesmcarthur has joined #openstack-infra | 03:41 | |
| *** IlyaG has quit IRC | 03:42 | |
| mordred | clarkb: yay. this means I get to add YET ANOTHER set of branching conditoins to create_image in shade | 03:42 |
| mordred | clarkb: because in v1 there is a direct option to tell glance to import from a url | 03:43 |
| mordred | which people have wanted, but we've never supported it in shade because there was no v2 equiv | 03:43 |
| clarkb | canyou only convert on import not upload? | 03:43 |
| mordred | except it seems there IS A v2 equiv | 03:43 |
| mordred | I do not know the answer to that | 03:43 |
| mordred | because tasks are undocumented | 03:43 |
| clarkb | right | 03:43 |
| mordred | clarkb: I'm going to poke at rax now | 03:44 |
| mordred | and also go read the glance source code | 03:44 |
| *** links has joined #openstack-infra | 03:45 | |
| mordred | | message | Unsupported element in image_properties, please consult the documentation. | | 03:46 |
| mordred | clarkb: there is a test case in glance that uses importing qcow2 into vhd format as the example payload | 03:50 |
| clarkb | wow its a thing you cant do with any toolI know of | 03:51 |
| mordred | well, I mean it shows the syntax for asking the cloud to do the conversion for you | 03:51 |
| mordred | simliar to the 'please convert this qcow2 image to raw format' which worked on vexxhost and ovh | 03:51 |
| *** piet has joined #openstack-infra | 03:51 | |
| mordred | clarkb: ok - I can't make it work on rax | 03:53 |
| mordred | clarkb: I'll ask rosmaita about it tomorrow | 03:54 |
| *** adrian_otto has joined #openstack-infra | 03:54 | |
| mordred | clarkb: are we still having rax issues? | 03:54 |
| clarkb | I dont think so | 03:54 |
| mordred | k | 03:55 |
| clarkb | new problem is with internap | 03:55 |
| *** baoli has joined #openstack-infra | 03:55 | |
| mordred | oh. do we know what's up therE? just can't upload? | 03:55 |
| clarkb | the token is only good for an hour and has to be valid at end of upload because glance uses it to talk to something else | 03:55 |
| clarkb | so leystoneauth is useless | 03:55 |
| mordred | oh. wow | 03:55 |
| fungi | basically today we got 3 of 4 images uploaded before the token expired | 03:56 |
| mordred | maybe we should get a new token before each upload? | 03:56 |
| fungi | the first 3 took around an hour and so the token was expired when or shortly after we tried to upload the 4th | 03:56 |
| *** Daisy_ has quit IRC | 03:56 | |
| clarkb | mordred: that was mgagne 's suggested worlaround | 03:57 |
| clarkb | this should also be fixed in glance | 03:57 |
| clarkb | but I dont have enough details to file a bug | 03:57 |
| mordred | yah | 03:57 |
| *** rguillebert has quit IRC | 03:57 | |
| fungi | trusts apparently fixes it, if you're on new enough openstack | 03:58 |
| *** johnsom_ is now known as johnsom | 03:59 | |
| mordred | oh good | 03:59 |
| *** baoli has quit IRC | 03:59 | |
| mordred | fungi, clarkb: image import-from-url works on internap | 04:00 |
| clarkb | we should make sure zanata and openstackid are working again soon with translations happening now and all that | 04:01 |
| mordred | ++ | 04:01 |
| *** fawadkhaliq has joined #openstack-infra | 04:01 | |
| mordred | import-from works on bluebox | 04:02 |
| mordred | so it seems that it works on all of our clouds | 04:02 |
| mordred | except rax | 04:02 |
| clarkb | osoc? | 04:02 |
| clarkb | er osic | 04:02 |
| mordred | osic is pretty recent, right? | 04:03 |
| mordred | (but testing) | 04:03 |
| clarkb | I dont know what they run but its a new cloud built by rax | 04:03 |
| mordred | clarkb: works on osic | 04:04 |
| clarkb | neat | 04:05 |
| mordred | clarkb: I will add support to shade for this | 04:05 |
| clarkb | unfortunately you cant kill tasks iirc | 04:06 |
| mordred | you cannot kill tasks - you can only poll for their status | 04:07 |
| StevenK | Huh, I have no idea what is going on with openstackid and zanata | 04:08 |
| clarkb | StevenK: well earlier today smarcet was doing things | 04:09 |
| clarkb | and there were reports of broken ipv6 | 04:09 |
| clarkb | I am guessing some.combination of these factors is related to the current behavior | 04:09 |
| clarkb | fungi ^ do you know what the things were? | 04:09 |
| fungi | clarkb: afaik implementing new features for summit apps support | 04:10 |
| fungi | pretty sure the ipv6-related dns issues were back on the 15th | 04:10 |
| fungi | i'm going to guess something in the recent openstackid patches has broken authenticating anything besides www.openstack.org | 04:11 |
| fungi | any idea if groups.o.o authentication is similarly broken? i think it's the only other thing using openstackid currently | 04:12 |
| fungi | unfortunately this is just going to make the tipit requests for ssh access to the production server all the more urgent | 04:12 |
| fungi | because apparently developing web apps isn't like developing software | 04:13 |
| clarkb | I havent seen any complaints for groups logins | 04:13 |
| *** Jeffrey4l has joined #openstack-infra | 04:13 | |
| clarkb | https://review.openstack.org/#/c/286703 | 04:14 |
| clarkb | while I have no problem with windows users writing windows docs, why a video? | 04:14 |
| clarkb | looks like the concerns have been raisedand asked for actual text docs to fit with everything else | 04:15 |
| *** Daisy has joined #openstack-infra | 04:17 | |
| fungi | i think it's that windows users don't read? | 04:20 |
| *** piet has quit IRC | 04:20 | |
| *** thorst has joined #openstack-infra | 04:21 | |
| *** adrian_otto has quit IRC | 04:24 | |
| *** asettle has joined #openstack-infra | 04:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:27 | |
| *** thorst has quit IRC | 04:28 | |
| *** treaki_ has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:31 | |
| *** asettle has quit IRC | 04:32 | |
| *** kzaitsev_mb has quit IRC | 04:32 | |
| *** treaki__ has quit IRC | 04:32 | |
| *** armax has quit IRC | 04:33 | |
| *** flepied has quit IRC | 04:33 | |
| *** bexelbie has quit IRC | 04:34 | |
| *** Daisy has quit IRC | 04:36 | |
| *** IlyaG has joined #openstack-infra | 04:38 | |
| *** vgridnev has joined #openstack-infra | 04:43 | |
| *** IlyaG has quit IRC | 04:43 | |
| *** fedexo has joined #openstack-infra | 04:47 | |
| *** flepied has joined #openstack-infra | 04:47 | |
| *** salv-orlando has joined #openstack-infra | 04:48 | |
| *** aspiers has joined #openstack-infra | 04:49 | |
| *** salv-orlando has quit IRC | 04:51 | |
| *** weshay has quit IRC | 04:55 | |
| *** armax has joined #openstack-infra | 04:59 | |
| *** gnuoy has quit IRC | 05:00 | |
| *** jamespage has quit IRC | 05:00 | |
| *** gnuoy has joined #openstack-infra | 05:00 | |
| *** jamespage has joined #openstack-infra | 05:01 | |
| *** Jeffrey4l has quit IRC | 05:03 | |
| *** Daisy has joined #openstack-infra | 05:03 | |
| *** verdurin has joined #openstack-infra | 05:07 | |
| *** ianychoi2 has joined #openstack-infra | 05:09 | |
| *** ianychoi has quit IRC | 05:09 | |
| armax | clarkb: ping? | 05:09 |
| prometheanfire | ubuntu-trusty seems to be missing uuidgen | 05:12 |
| prometheanfire | https://review.openstack.org/#/c/290894/2 | 05:12 |
| prometheanfire | should be fixed in this review though | 05:12 |
| *** adrian_otto has joined #openstack-infra | 05:13 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add testing for the Gentoo element https://review.openstack.org/290894 | 05:13 |
| *** markvoelker has joined #openstack-infra | 05:15 | |
| *** baoli has joined #openstack-infra | 05:15 | |
| *** adrian_otto has quit IRC | 05:16 | |
| *** adrian_otto has joined #openstack-infra | 05:16 | |
| *** bexelbie has joined #openstack-infra | 05:16 | |
| *** ianychoi2 is now known as ianychoi | 05:18 | |
| *** baoli has quit IRC | 05:20 | |
| *** markvoelker has quit IRC | 05:21 | |
| *** fawadkhaliq has quit IRC | 05:21 | |
| *** harlowja_at_home has quit IRC | 05:24 | |
| *** harlowja_at_home has joined #openstack-infra | 05:25 | |
| *** thorst has joined #openstack-infra | 05:26 | |
| *** _nadya_ has joined #openstack-infra | 05:27 | |
| *** asselin_ has joined #openstack-infra | 05:32 | |
| *** thorst has quit IRC | 05:33 | |
| *** asselin__ has quit IRC | 05:33 | |
| *** achanda has quit IRC | 05:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: [WIP] Bring up glean before network.service https://review.openstack.org/293826 | 05:42 |
| *** asettle has joined #openstack-infra | 05:43 | |
| *** sdake has joined #openstack-infra | 05:44 | |
| *** [HeOS] has quit IRC | 05:45 | |
| *** yamahata has quit IRC | 05:45 | |
| *** sc68cal has quit IRC | 05:45 | |
| *** HeOS has joined #openstack-infra | 05:45 | |
| *** sdake_ has joined #openstack-infra | 05:45 | |
| *** abregman has joined #openstack-infra | 05:46 | |
| *** sdake has quit IRC | 05:49 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms https://review.openstack.org/293731 | 05:49 |
| *** MarkAtwood has quit IRC | 05:50 | |
| *** sdake_ is now known as sdake | 05:51 | |
| ianw | prometheanfire: can you put that in a separate change and i can quick approve it | 05:56 |
| prometheanfire | ianw: the cleanup? | 05:58 |
| ianw | the new package for ubuntu | 05:58 |
| prometheanfire | oh :( | 05:58 |
| ianw | it's breaking generically, right? | 05:59 |
| prometheanfire | I'm actually kinda far from my gpg/ssh key | 05:59 |
| prometheanfire | it is | 05:59 |
| prometheanfire | it just needs that package (I think) | 05:59 |
| ianw | yes, devstack hit the same issue. i'll push it in | 06:00 |
| prometheanfire | I'm fine if you take that fix | 06:00 |
| prometheanfire | k | 06:00 |
| prometheanfire | nn | 06:00 |
| *** Qiming_ has joined #openstack-infra | 06:00 | |
| prometheanfire | not gonna stay up as early as last night | 06:00 |
| *** Qiming has quit IRC | 06:01 | |
| *** Daisy has quit IRC | 06:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add uuid-runtime for Trusty https://review.openstack.org/293833 | 06:01 |
| *** Daisy has joined #openstack-infra | 06:02 | |
| *** esikachev has joined #openstack-infra | 06:04 | |
| *** rlandy has quit IRC | 06:06 | |
| *** chlong has quit IRC | 06:08 | |
| *** esikachev has quit IRC | 06:09 | |
| armax | I wonder if any of you has any recommendation on how to add a missing package into the image to allow neutron tests to work correctly | 06:09 |
| armax | it seems that after change https://review.openstack.org/#/c/292573/ | 06:09 |
| armax | the netcat-openbsd package is no longer available | 06:09 |
| armax | and this breaks a neutron job | 06:10 |
| armax | this is from a good run: | 06:10 |
| armax | http://logs.openstack.org/39/293239/3/check/gate-neutron-dsvm-functional/f1284e9/logs/dpkg-l.txt.gz | 06:10 |
| armax | this is from a bad run | 06:10 |
| armax | http://logs.openstack.org/87/293587/1/check/gate-neutron-dsvm-functional/53d6bee/logs/dpkg-l.txt.gz | 06:10 |
| *** _nadya_ has quit IRC | 06:10 | |
| armax | I’ll try poking at devstack itself, but a heads-up on this would be nice | 06:11 |
| *** oanson has quit IRC | 06:12 | |
| *** salv-orlando has joined #openstack-infra | 06:13 | |
| *** Daisy has quit IRC | 06:17 | |
| *** roxanagh_ has joined #openstack-infra | 06:17 | |
| ianw | armax: you can add netcat-openbsd to files/debs/neutron in devsatck | 06:17 |
| *** oanson has joined #openstack-infra | 06:18 | |
| armax | ianw: that’s what I am about to do | 06:18 |
| armax | ianw: do you think that’s the right thing to do ? | 06:18 |
| ianw | switching these nodes is a bit of a pain, but OTOH us getting the dependencies explicitly listed is a good idea | 06:18 |
| armax | ianw: if so, I am happy to proceed :) | 06:18 |
| armax | ianw: ack | 06:18 |
| ianw | armax: yes. i can quick approve that if it gets it going | 06:18 |
| ianw | i did similar for uuidgen | 06:18 |
| armax | ianw: posting quickly hang on | 06:19 |
| *** chlong has joined #openstack-infra | 06:21 | |
| *** roxanagh_ has quit IRC | 06:22 | |
| armax | ianw: https://review.openstack.org/#/c/293837/ | 06:22 |
| armax | johnsom: ^ | 06:22 |
| armax | ianw: I wonder if this must be backported | 06:22 |
| armax | I suppose it must, no? | 06:22 |
| ianw | probably, yes | 06:23 |
| *** salv-orlando has quit IRC | 06:23 | |
| armax | though I am not quite sure how we’re broken on neutron stable/mitaka and yet the stable/mitaka branch is not yet open for devstack | 06:23 |
| ianw | hang on, hang, that's functional tests | 06:24 |
| armax | ianw: right, I think we can try some other way | 06:28 |
| *** kzaitsev_mb has joined #openstack-infra | 06:29 | |
| armax | ianw: I am still wrapping my head around this | 06:29 |
| ianw | armax: actually, no, that's the right fix -> https://git.openstack.org/cgit/openstack/neutron/tree/tools/configure_for_func_testing.sh#n85 | 06:29 |
| ianw | neutron functional tests uses the devstack package list | 06:29 |
| armax | that’s where I am right now, but as you can see, we don’t maintain a list of deps in tree | 06:30 |
| armax | or am I supposed to add that deps to a different file in devstack | 06:30 |
| armax | ? | 06:30 |
| *** thorst has joined #openstack-infra | 06:31 | |
| ianw | armax: so it seems like the functional tests use the devstack package list, but also has a "post" section where it installs a few other things | 06:32 |
| armax | ianw: do you think it’s best if we kept a list for test deps in the neutron tree? | 06:32 |
| ianw | armax: i think you kind of have it, see my second comment | 06:32 |
| *** Jeffrey4l has joined #openstack-infra | 06:32 | |
| ianw | https://git.openstack.org/cgit/openstack/neutron/tree/tools/configure_for_func_testing.sh#n217 | 06:32 |
| *** flepied has quit IRC | 06:33 | |
| *** BigWillie has joined #openstack-infra | 06:33 | |
| ianw | so there's a few too many choices to just push in a quick fix for this at this point. i think it mostly depends on what neutron wants to do | 06:33 |
| armax | ianw: I think this lies with the functional testing setup | 06:34 |
| *** ociuhandu has quit IRC | 06:34 | |
| ianw | armax: yes, i'd propose it in that fn at line 217 there i think | 06:34 |
| armax | ianw: if we can help it | 06:34 |
| *** BigWillie has quit IRC | 06:34 | |
| *** kzaitsev_mb has quit IRC | 06:35 | |
| armax | ianw: let me abandon the devstack fix | 06:35 |
| armax | and thanks for the extra pair of eyes | 06:36 |
| *** ociuhandu has joined #openstack-infra | 06:36 | |
| ianw | armax: heh, cool, i have to go now too. probably not till us east time does devstack people come back | 06:36 |
| tobiash | jhesketh: would you agree to make the parameters of GerritWatcher.__init__ mandatory? This hides forgotten configuration values of this class (see https://review.openstack.org/#/c/238988/11 again the keepalive stuff...) | 06:36 |
| armax | ianw: thanks again! | 06:37 |
| *** thorst has quit IRC | 06:38 | |
| jhesketh | tobiash: I think having a default on the parameter is okay and in line with the other params | 06:40 |
| jhesketh | tobiash: it's only called from the GerritConnection (afaik) so ensuring the config value is passed through correctly there there shouldn't be any problems | 06:40 |
| *** fedexo has quit IRC | 06:41 | |
| tobiash | jhesketh: As you wish, but making all of these parameters mandatory would make this part more robust against future refactoring mistakes as this would automatically enforce that all of the the config values are passed through. | 06:45 |
| jhesketh | sure, I think both ways are fine... it depends on whether we expect GerritWatcher to be used elsewhere and if it were if there was a need for sane defaults on the parameters (and thereofre an expectation that not every instance of GerritWatcher is being configured by a config file) | 06:47 |
| jhesketh | however since it is only the one place, removing the defaults is fine and possibly a good idea | 06:47 |
| jhesketh | tobiash: in other words, feel free to change them if you'd like | 06:47 |
| tobiash | :) | 06:48 |
| jhesketh | the only tricky part is that we don't want too much into master while v3 is being worked on. But since that line is changing anyway it shouldn't hurt | 06:48 |
| *** kingia_ has quit IRC | 06:48 | |
| tobiash | well, this default parameters stuff is not critical and could be done directly on the v3 branch | 06:49 |
| tobiash | jhesketh: btw, did you take the ownership of this keepalive patch? | 06:49 |
| tobiash | saw that the latest versions are from you | 06:50 |
| *** kingia has joined #openstack-infra | 06:50 | |
| jhesketh | tobiash: not ownership, just helping rebasing it when doing the connection work | 06:52 |
| tobiash | ah, ok | 06:52 |
| *** markvoelker has joined #openstack-infra | 06:52 | |
| openstackgerrit | Travis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs https://review.openstack.org/293848 | 06:55 |
| *** akaszuba has joined #openstack-infra | 06:57 | |
| *** markvoelker has quit IRC | 06:58 | |
| *** jaosorior has joined #openstack-infra | 06:58 | |
| *** Daisy has joined #openstack-infra | 06:59 | |
| *** HeOS has quit IRC | 07:00 | |
| *** _nadya_ has joined #openstack-infra | 07:01 | |
| *** HeOS has joined #openstack-infra | 07:01 | |
| *** HeOS has quit IRC | 07:02 | |
| openstackgerrit | Travis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs https://review.openstack.org/293848 | 07:02 |
| *** Daisy has quit IRC | 07:03 | |
| *** HeOS has joined #openstack-infra | 07:03 | |
| *** esikachev has joined #openstack-infra | 07:08 | |
| *** alex_xu has quit IRC | 07:10 | |
| *** jiangliucheng has quit IRC | 07:12 | |
| *** jiangliucheng has joined #openstack-infra | 07:12 | |
| *** alex_xu has joined #openstack-infra | 07:12 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Enable voting for gate jobs in sahara-tests https://review.openstack.org/293464 | 07:15 |
| *** mikelk has joined #openstack-infra | 07:15 | |
| *** tesseract has joined #openstack-infra | 07:15 | |
| *** asettle has quit IRC | 07:15 | |
| *** tesseract is now known as Guest57182 | 07:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Use pip installed dib for functional tests https://review.openstack.org/293854 | 07:16 |
| ianw | greghaynes / SpamapS : so i think that will be required to get dib functional tests working again | 07:16 |
| ianw | ^ | 07:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 07:17 |
| *** vgridnev has quit IRC | 07:17 | |
| *** _nadya_ has quit IRC | 07:18 | |
| *** Jeffrey4l has quit IRC | 07:22 | |
| *** jogo has quit IRC | 07:22 | |
| *** sridhar_ram has joined #openstack-infra | 07:23 | |
| ianw | AJaeger: there seems to be quite a bit of fallout from these changes, i wonder if we can do things in a more controlled fashion? | 07:24 |
| ianw | i don't know what that is, although i agree we end up in a better place having dependencies specified in the long run | 07:24 |
| *** claudiub|2 has joined #openstack-infra | 07:24 | |
| *** jogo has joined #openstack-infra | 07:25 | |
| *** jogo has joined #openstack-infra | 07:25 | |
| *** zeih has joined #openstack-infra | 07:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:30 | |
| *** gongysh has joined #openstack-infra | 07:31 | |
| *** scheuran has joined #openstack-infra | 07:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Use pip installed dib for functional tests https://review.openstack.org/293854 | 07:32 |
| *** salv-orlando has joined #openstack-infra | 07:32 | |
| *** _nadya_ has joined #openstack-infra | 07:32 | |
| *** mikelk has quit IRC | 07:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: uc: openstackdocstheme==1.3.0 https://review.openstack.org/293859 | 07:34 |
| *** Daisy has joined #openstack-infra | 07:34 | |
| *** kzaitsev_mb has quit IRC | 07:35 | |
| *** thorst has joined #openstack-infra | 07:36 | |
| *** rcernin has joined #openstack-infra | 07:39 | |
| *** salv-orlando has quit IRC | 07:40 | |
| *** ihrachys has joined #openstack-infra | 07:41 | |
| *** kingia has quit IRC | 07:42 | |
| *** thorst has quit IRC | 07:43 | |
| *** flepied has joined #openstack-infra | 07:44 | |
| *** kingia has joined #openstack-infra | 07:44 | |
| *** jiangliucheng has quit IRC | 07:47 | |
| *** k4n0 has joined #openstack-infra | 07:48 | |
| *** Daisy has quit IRC | 07:48 | |
| *** Daisy has joined #openstack-infra | 07:49 | |
| *** camunoz has quit IRC | 07:50 | |
| *** Daisy has quit IRC | 07:53 | |
| *** jlanoux has joined #openstack-infra | 07:54 | |
| *** chlong has quit IRC | 07:57 | |
| AJaeger | ianw: the bare-trusty went so smooth and the initial first test and first conversion of devstack-trusty that clarkb converted them all in one. | 07:59 |
| AJaeger | ianw: We expected less problems here ;( | 07:59 |
| *** HeOS has quit IRC | 08:00 | |
| abregman | can anyone give me newbie tasks in openstack-infra projects? :( | 08:00 |
| *** cody-somerville has quit IRC | 08:01 | |
| *** jlanoux has quit IRC | 08:03 | |
| *** Daisy has joined #openstack-infra | 08:04 | |
| *** jlanoux has joined #openstack-infra | 08:04 | |
| *** HeOS has joined #openstack-infra | 08:05 | |
| AJaeger | abregman: start with reviewing ;) | 08:05 |
| *** k4n0 has quit IRC | 08:06 | |
| abregman | AJaeger: any specific projects? | 08:06 |
| abregman | is there even a way to filter only openstack-infra reviews? | 08:06 |
| *** hichihara has quit IRC | 08:07 | |
| *** amrith is now known as _amrith_ | 08:10 | |
| AJaeger | abregman: you can filter using the search box, e.g. enter: "project:openstack-infra/project-config" and add also "status:open" - this will give you https://review.openstack.org/#/q/project:openstack-infra/project-config+status:open | 08:11 |
| AJaeger | You can also filter for all projects like "projects:openstack-infra/" | 08:11 |
| abregman | AJaeger: ack. I'll do that. thanks! | 08:11 |
| AJaeger | abregman: where to start? It depends on what you want to do... | 08:11 |
| abregman | Ansible, python.. | 08:12 |
| *** mikelk has joined #openstack-infra | 08:12 | |
| AJaeger | abregman: our current specs are up at http://specs.openstack.org/openstack-infra/infra-specs/ | 08:13 |
| AJaeger | and you can even filter on those if you're interested in a topic for review like https://review.openstack.org/#/q/status:open+++topic:i18n-checksite | 08:13 |
| AJaeger | abregman: for ansible, system-config is more appropriate | 08:13 |
| AJaeger | I suggest you talk with rest of infra team during US office hours | 08:14 |
| AJaeger | others might give you some more pointers on newbie tasks | 08:14 |
| AJaeger | abregman: Sorry, I forgot: Glad to hear that you're interested! | 08:15 |
| *** zeih has quit IRC | 08:15 | |
| abregman | AJaeger: thanks for the tips :) I'll ask again during US office hours and look on some changes in gerrit | 08:15 |
| AJaeger | abregman: http://docs.openstack.org/infra/ is our central docu starting page | 08:15 |
| *** vgridnev has joined #openstack-infra | 08:16 | |
| Daisy | AJaeger: Hi. Just let you know. I'm not able to login to translate website. So I didn't make any versions yet. | 08:17 |
| *** sridhar_ram1 has joined #openstack-infra | 08:17 | |
| AJaeger | Daisy: I've seen - yeah this is really bad timing ;( | 08:17 |
| Daisy | AJaeger: yeah. Do you have any ideas about the problem ? | 08:18 |
| AJaeger | openstackid integration is broken | 08:18 |
| *** zeih has joined #openstack-infra | 08:18 | |
| AJaeger | Hope they get it fixed quickly | 08:18 |
| Daisy | AJaeger: Translators can login to www.openstack.com with Openstack ID, but cannot login to Zanata. | 08:19 |
| *** sridhar_ram has quit IRC | 08:19 | |
| AJaeger | if it's not working tomorrow morning, please come back directly | 08:19 |
| *** sflanigan has quit IRC | 08:19 | |
| * AJaeger will try to ping people to fix it but haven't seen anyone around yet | 08:19 | |
| zigo | AJaeger: Hi there. When I a job is ran, what parameter does it get in the env? Namely, do I get the name of the gerrit repo? How does it get the CR # and such? | 08:20 |
| Daisy | AJaeger: I think one day can be tolerant. I don't think we can wait for two days. It's busy time for translators. | 08:20 |
| *** kingia_ has joined #openstack-infra | 08:20 | |
| AJaeger | zigo, you can check on jenkins what parameters it gets and check your log file as well. | 08:22 |
| AJaeger | Daisy: I know, I fear both of us need to ping here ;( | 08:22 |
| *** kingia has quit IRC | 08:23 | |
| *** salv-orlando has joined #openstack-infra | 08:23 | |
| Daisy | AJaeger: understand. I will ping when people wake up. | 08:23 |
| AJaeger | zigo: We normally set jobs up so that they have a repo that contains the change applied. | 08:23 |
| *** pcaruana has joined #openstack-infra | 08:24 | |
| Daisy | AJaeger: hope it's not too difficult and they can fix it today. | 08:24 |
| AJaeger | +1 | 08:24 |
| *** nmagnezi has joined #openstack-infra | 08:25 | |
| AJaeger | infra-root: according to status.o.o/zuul the periodic queue is not moving forward, did we lose the proposal slave? | 08:26 |
| *** esikachev has quit IRC | 08:27 | |
| *** arxcruz has joined #openstack-infra | 08:27 | |
| *** yamamoto_ has joined #openstack-infra | 08:29 | |
| * AJaeger will be back later | 08:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:31 | |
| zigo | AJaeger: I don't have a job yet, so I don't really know where to look. | 08:35 |
| *** fhubik has joined #openstack-infra | 08:35 | |
| *** esikachev has joined #openstack-infra | 08:36 | |
| zigo | AJaeger: I believe I understood the thingy with jenkins/jobs/* and jenkins/scripts/* stuff. | 08:36 |
| *** ifarkas has joined #openstack-infra | 08:36 | |
| *** kzaitsev_mb has quit IRC | 08:36 | |
| zigo | AJaeger: Now, I'm trying to understand what will be passed to my script... | 08:36 |
| *** vincentll has joined #openstack-infra | 08:36 | |
| ogelbukh_ | hi team, looks like this change has sign offs needed: https://review.openstack.org/#/c/286137/ | 08:39 |
| ogelbukh_ | or this is the wrong place to ask for reviews? | 08:39 |
| *** sshnaidm has quit IRC | 08:40 | |
| *** johnsom has quit IRC | 08:40 | |
| *** mrmartin has joined #openstack-infra | 08:40 | |
| *** thorst has joined #openstack-infra | 08:40 | |
| xiangxinyong | AJaeger: ping | 08:41 |
| *** johnsom has joined #openstack-infra | 08:41 | |
| xiangxinyong | AJaeger: could i ask a question about the ci? | 08:42 |
| *** sdake has quit IRC | 08:43 | |
| *** SumitNaiksatam has quit IRC | 08:43 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 08:43 | |
| *** sdake has joined #openstack-infra | 08:43 | |
| zigo | AJaeger: Under what users are script executed? | 08:44 |
| zigo | root? | 08:44 |
| *** rossella_s has joined #openstack-infra | 08:44 | |
| *** fhubik has quit IRC | 08:45 | |
| *** hashar has joined #openstack-infra | 08:48 | |
| *** thorst has quit IRC | 08:48 | |
| *** jcoufal has joined #openstack-infra | 08:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:48 | |
| *** fhubik has joined #openstack-infra | 08:50 | |
| *** sdake has quit IRC | 08:50 | |
| *** cody-somerville has joined #openstack-infra | 08:52 | |
| *** links has quit IRC | 08:53 | |
| *** bexelbie has quit IRC | 08:53 | |
| *** bexelbie has joined #openstack-infra | 08:53 | |
| *** jordanP has joined #openstack-infra | 08:55 | |
| *** gildub has quit IRC | 08:58 | |
| *** dingyichen has quit IRC | 08:59 | |
| *** cody-somerville has quit IRC | 09:01 | |
| *** cody-somerville has joined #openstack-infra | 09:02 | |
| *** ihrachys has quit IRC | 09:02 | |
| *** rossella_s has quit IRC | 09:03 | |
| *** rossella_s has joined #openstack-infra | 09:03 | |
| *** onovy has quit IRC | 09:07 | |
| *** peterlisak has quit IRC | 09:07 | |
| *** salv-orlando has quit IRC | 09:07 | |
| openstackgerrit | Hua Wang proposed openstack-infra/devstack-gate: SERVICE_HOST should not be 127.0.0.1 https://review.openstack.org/293890 | 09:08 |
| *** zhurong has quit IRC | 09:10 | |
| *** fhubik is now known as fhubik_brb | 09:10 | |
| *** fhubik_brb is now known as fhubik | 09:10 | |
| yolanda | good mornign | 09:12 |
| yolanda | AJaeger, i'll take a look at zuuo | 09:12 |
| yolanda | zuul | 09:12 |
| *** d0ugal has quit IRC | 09:13 | |
| *** d0ugal has joined #openstack-infra | 09:14 | |
| *** akaszuba has quit IRC | 09:16 | |
| *** akaszuba has joined #openstack-infra | 09:19 | |
| *** asettle has joined #openstack-infra | 09:19 | |
| yolanda | AJaeger so yes, proposal is offline, i'm checking the reasons now | 09:22 |
| yolanda | mm | 09:22 |
| yolanda | Disconnected by anonymous : Offline due to Gearman request | 09:22 |
| yolanda | there should be some job incorrectly configured | 09:22 |
| yolanda | i put back online now | 09:23 |
| yolanda | AJaeger, according to build history, the latest job run was https://jenkins.openstack.org/job/neutron-propose-translation-update-mitaka/ | 09:23 |
| rcarrillocruz | hmm | 09:26 |
| rcarrillocruz | AJaeger, yolanda: https://review.openstack.org/#/c/242161/9 | 09:26 |
| *** Daisy has quit IRC | 09:26 | |
| rcarrillocruz | was it agreed to install npm as a dep for all slaves, rather than downloading per-job basis | 09:26 |
| rcarrillocruz | ? | 09:26 |
| *** jistr has joined #openstack-infra | 09:26 | |
| *** Daisy has joined #openstack-infra | 09:26 | |
| *** Daisy_ has joined #openstack-infra | 09:28 | |
| yolanda | i don't know, we'll need to check with another roots in US | 09:28 |
| rcarrillocruz | i mean, functionally is correct, but would like to understand if that was agreed upon prior to giving +1 | 09:29 |
| *** kzaitsev_mb has quit IRC | 09:29 | |
| rcarrillocruz | sounds like the typical thing that was at least discussed here on in the infra meeting | 09:30 |
| yolanda | i'd say that this is done based on agreement, but cannot ensure 100% | 09:30 |
| *** onovy has joined #openstack-infra | 09:31 | |
| *** peterlisak has joined #openstack-infra | 09:31 | |
| *** Daisy has quit IRC | 09:31 | |
| *** samuelBartel has joined #openstack-infra | 09:32 | |
| *** sshnaidm has joined #openstack-infra | 09:34 | |
| *** fverbs has joined #openstack-infra | 09:37 | |
| yolanda | AJaeger, i'd say that the propose-translation-update-mitaka jobs are breaking proposal, going to take a look | 09:38 |
| *** esikachev has quit IRC | 09:40 | |
| *** achanda has joined #openstack-infra | 09:44 | |
| *** matrohon has joined #openstack-infra | 09:45 | |
| *** achanda has quit IRC | 09:45 | |
| *** thorst has joined #openstack-infra | 09:45 | |
| *** esikachev has joined #openstack-infra | 09:46 | |
| *** sridhar_ram1 has quit IRC | 09:46 | |
| *** achanda has joined #openstack-infra | 09:46 | |
| *** achanda has quit IRC | 09:46 | |
| *** _degorenko|afk is now known as degorenko | 09:47 | |
| *** Daisy_ has quit IRC | 09:47 | |
| tobiash | jhesketh: just curious, how many zuul-mergers do you have running for the openstack infrastructure? | 09:48 |
| *** Daisy has joined #openstack-infra | 09:48 | |
| rcarrillocruz | tobiash: http://puppetdb.openstack.org/nodes | 09:50 |
| rcarrillocruz | zm? | 09:50 |
| rcarrillocruz | if memory serves, in gozer we had reusable-node for jobs using static slaves | 09:51 |
| rcarrillocruz | yolanda: ^ | 09:51 |
| *** sdague has joined #openstack-infra | 09:51 | |
| yolanda | i'm looking at upstream and i cannot see that reusable-node being used | 09:51 |
| yolanda | anywhere in layout.yaml | 09:51 |
| yolanda | so not sure if that's done in a different way | 09:52 |
| *** kelepirci has joined #openstack-infra | 09:52 | |
| kelepirci | hello all | 09:52 |
| *** Daisy has quit IRC | 09:52 | |
| *** thorst has quit IRC | 09:52 | |
| rcarrillocruz | hmm | 09:52 |
| rcarrillocruz | yeah | 09:52 |
| rcarrillocruz | grep does not show that on zuul's code | 09:53 |
| yolanda | there is something wrong in the mitake jobs, the other translation jobs are fine | 09:53 |
| rcarrillocruz | ... | 09:53 |
| rcarrillocruz | that's strange | 09:53 |
| rcarrillocruz | let's see the git log | 09:53 |
| yolanda | but when a mitaka-update job runs, it turns proposal off | 09:53 |
| yolanda | a8317f435657c7250312c32fb99d0b077c934086 | 09:54 |
| yolanda | but i don't see anything different than other translation jobs | 09:55 |
| rcarrillocruz | well | 09:55 |
| rcarrillocruz | https://review.openstack.org/#/c/74888/ | 09:55 |
| rcarrillocruz | they used that flag at some point | 09:55 |
| rcarrillocruz | so if that's not on layout.yaml anymore | 09:55 |
| rcarrillocruz | something changed | 09:55 |
| rcarrillocruz | but def. they used it | 09:56 |
| rcarrillocruz | and for proposal | 09:56 |
| yolanda | no parameter-function anymore... | 09:56 |
| rcarrillocruz | parameter-function does exist on zuul's code tho | 09:57 |
| yolanda | so again , https://jenkins.openstack.org/job/heat-propose-translation-update-mitaka/ turned proposal offline. Something is wrong with these jobs | 09:57 |
| rcarrillocruz | ... | 09:57 |
| *** dims has joined #openstack-infra | 09:57 | |
| *** dguitarbite has joined #openstack-infra | 10:00 | |
| yolanda | so looking at job env vars, i see offline_node_when_complete set to 1 in the jobs that fails. The other jobs that run, are not sending that env var | 10:00 |
| dguitarbite | hello, I am not able to login to gerrit from review.openstack.org for some reason | 10:00 |
| rcarrillocruz | ah right | 10:00 |
| yolanda | but i don't know how that is configured | 10:00 |
| rcarrillocruz | so | 10:00 |
| rcarrillocruz | the function | 10:00 |
| rcarrillocruz | reusable_node | 10:00 |
| rcarrillocruz | is defined on project_config | 10:00 |
| yolanda | oh | 10:01 |
| yolanda | i see | 10:01 |
| yolanda | based on a regex | 10:01 |
| yolanda | rcarrillocruz, do you want to send a fix to add mitaka? | 10:01 |
| *** _nadya_ has quit IRC | 10:01 | |
| rcarrillocruz | sure, where's the line of the regex | 10:02 |
| *** jamesmcarthur has joined #openstack-infra | 10:03 | |
| *** nadya has joined #openstack-infra | 10:04 | |
| *** tpsilva has joined #openstack-infra | 10:04 | |
| *** jamesmcarthur has quit IRC | 10:08 | |
| yolanda | so zuul/openstack_functions, you'll see the proposal_re | 10:08 |
| yolanda | need to add update-mitaka there | 10:09 |
| rcarrillocruz | yup, grepped, thx | 10:09 |
| rcarrillocruz | i'll push shortly | 10:09 |
| yolanda | thanks | 10:11 |
| *** Qiming_ has quit IRC | 10:11 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Add mitaka to proposal regex https://review.openstack.org/293923 | 10:14 |
| *** daemontool has joined #openstack-infra | 10:14 | |
| *** jaosorior has quit IRC | 10:14 | |
| *** jaosorior has joined #openstack-infra | 10:15 | |
| rcarrillocruz | AJaeger, yolanda: ^ | 10:15 |
| *** yamamoto_ has quit IRC | 10:17 | |
| kelepirci | I am having some problem with loging in to https://wiki.openstack.org/wiki/Main_Page | 10:17 |
| kelepirci | where should I get support fot this? | 10:17 |
| yolanda | kelepirci, what problem do you have? sso problems? | 10:17 |
| *** placko_ has joined #openstack-infra | 10:18 | |
| kelepirci | I am trying to login to wiki with OpenID | 10:18 |
| kelepirci | the page redirecting me to ubuntu page and I alsrady have laucnhpad account. | 10:18 |
| *** esikachev has quit IRC | 10:18 | |
| kelepirci | it asks permission and I confirm. | 10:18 |
| kelepirci | after that I am see a page Log in / create account with open id | 10:19 |
| kelepirci | When I click on the button. nothings happens | 10:19 |
| kelepirci | than I stuck with this URL https://wiki.openstack.org/wiki/Special:OpenIDLogin/ChooseName | 10:20 |
| yolanda | going to check myself | 10:21 |
| mrmartin | yolanda: wiki.o.o is down? | 10:21 |
| yolanda | i cannot enter | 10:22 |
| yolanda | it takes long time | 10:22 |
| mrmartin | high load? | 10:22 |
| mrmartin | I'm trying to login too | 10:22 |
| *** tzn has joined #openstack-infra | 10:22 | |
| mrmartin | 28.8 load | 10:23 |
| yolanda | i could login successfully and is faster now | 10:23 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3 https://review.openstack.org/246017 | 10:23 |
| *** ociuhandu has quit IRC | 10:24 | |
| *** rguillebert has joined #openstack-infra | 10:25 | |
| yolanda | kelepirci, can you try again? | 10:25 |
| mrmartin | ok the load is much better now | 10:26 |
| mrmartin | I can imagine it got another attack. | 10:26 |
| kelepirci | let me try | 10:27 |
| *** yamamoto has joined #openstack-infra | 10:27 | |
| kelepirci | nope samethings happens | 10:27 |
| *** bradjones has quit IRC | 10:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/293933 | 10:28 |
| yolanda | kelepirci, what's your ID for launchpad, so i can check logs? | 10:28 |
| yolanda | i see several requests on logs | 10:28 |
| *** links has joined #openstack-infra | 10:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:29 | |
| *** daemontool has quit IRC | 10:31 | |
| *** adrian_otto has quit IRC | 10:31 | |
| yolanda | AJaeger, or project-config cores, https://review.openstack.org/#/c/293923 will fix the proposal slave problem | 10:33 |
| krotscheck | Anyone around who knows things about cloud-config? | 10:34 |
| *** kzaitsev_mb has quit IRC | 10:34 | |
| * krotscheck is curious about the format, because it might be a good data structure by which he can configure javascript webclient cloud apps. | 10:34 | |
| kelepirci | All users need a nickname; you can choose one from the options below. | 10:34 |
| kelepirci | I am getting following error | 10:34 |
| krotscheck | mordred: When you wake up, I'd like to pick your brain about cloudconfig things. | 10:36 |
| *** bradjones has joined #openstack-infra | 10:36 | |
| *** bradjones has quit IRC | 10:36 | |
| *** bradjones has joined #openstack-infra | 10:36 | |
| kelepirci | I think we are hitting following bug | 10:36 |
| kelepirci | http://wikitravel.org/shared/Tech:Can%27t_Complete_OpenID_login | 10:37 |
| kelepirci | where should I a report this bug? | 10:37 |
| yolanda | kelepirci, you can file bugs on https://storyboard.openstack.org/#!/project_group/55 | 10:38 |
| yolanda | maybe pabelanger can help you more when he comes online | 10:39 |
| kelepirci | ok than I will mage the bug report and wait for pabelanger | 10:40 |
| rcarrillocruz | mordred: need to chat with you about ansible modules and region handling, gimme a ping when you're around | 10:43 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: add uuid-runtime to base image builds https://review.openstack.org/293942 | 10:44 |
| *** BobBall_AWOL is now known as BobBall | 10:45 | |
| *** e0ne has joined #openstack-infra | 10:50 | |
| *** thorst has joined #openstack-infra | 10:51 | |
| *** esikachev has joined #openstack-infra | 10:51 | |
| AJaeger | yolanda, rcarrillocruz thanks a lot for fixing the proposal problem that I introduced ;( | 10:52 |
| rcarrillocruz | np | 10:53 |
| yolanda | AJaeger np | 10:53 |
| *** Jeffrey4l has joined #openstack-infra | 10:53 | |
| *** placko_ has quit IRC | 10:55 | |
| AJaeger | kelepirci: is this the first time you try to login to the wiki? We disabled new account creation it at one point due to spam - not sure whether it's enabled again | 10:55 |
| *** placko_ has joined #openstack-infra | 10:55 | |
| openstackgerrit | Merged openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms https://review.openstack.org/293731 | 10:55 |
| *** esikachev has quit IRC | 10:56 | |
| *** ociuhandu has joined #openstack-infra | 10:57 | |
| *** asettle has quit IRC | 10:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:57 | |
| *** thorst has quit IRC | 10:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Add mitaka to proposal regex https://review.openstack.org/293923 | 10:59 |
| *** zeih has quit IRC | 11:01 | |
| *** kzaitsev_mb has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Edit task notes from board https://review.openstack.org/291755 | 11:02 |
| *** cfarquhar has quit IRC | 11:03 | |
| *** acabot has quit IRC | 11:03 | |
| *** ihrachys has joined #openstack-infra | 11:04 | |
| *** kgiusti has left #openstack-infra | 11:07 | |
| *** salv-orlando has joined #openstack-infra | 11:07 | |
| *** dguitarbite has quit IRC | 11:07 | |
| *** Qiming has joined #openstack-infra | 11:12 | |
| *** boris-42 has joined #openstack-infra | 11:14 | |
| *** salv-orlando has quit IRC | 11:18 | |
| *** vgridnev has quit IRC | 11:19 | |
| *** smarcet has joined #openstack-infra | 11:20 | |
| yolanda | AJaeger, are you ok with approving https://review.openstack.org/#/c/293719/ ? | 11:21 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file https://review.openstack.org/293586 | 11:21 |
| yolanda | puppet jobs to ubuntu-trusty | 11:21 |
| *** gongysh has quit IRC | 11:23 | |
| smarcet | fungi: yolanda : morning , when u get a change please review https://review.openstack.org/#/c/293586/ , thx ! | 11:25 |
| doug-fish | AJaeger: Hi - I see that Zanata is still unavailable this morning - any update? | 11:29 |
| AJaeger | yolanda: yes, I am - but it needs probably manual jenkins update | 11:29 |
| AJaeger | doug-fish: let's ask smarcet ^ | 11:29 |
| yolanda | i haven't checked how many jobs, AJaeger. What's the number? | 11:29 |
| AJaeger | yolanda: lots of ;( It did not download the log file completely for me... | 11:30 |
| yolanda | risky... | 11:30 |
| AJaeger | smarcet: will 293586 fix openstackid for Zanata? | 11:30 |
| yolanda | i put a +2 but not approved | 11:30 |
| pabelanger | kelepirci: yolanda: Ya, we've disabled new users on the wiki for the moment. We have a patch up to re-enable it, we need 287232 to land | 11:30 |
| pabelanger | also, morning | 11:30 |
| yolanda | good morning pabelanger | 11:31 |
| *** acabot has joined #openstack-infra | 11:31 | |
| pabelanger | yolanda: AJaeger: do you have a moment to review 293767? Adds puppet-apply job for debian-jessie, which was brought online last night | 11:32 |
| pabelanger | and experimental job | 11:33 |
| AJaeger | pabelanger: the translation check site needs someone to help moving forward, is this something you have time for? See https://review.openstack.org/#/q/status:open+++topic:i18n-checksite - might be quite involved but important for release | 11:33 |
| AJaeger | pabelanger: will review shortly | 11:33 |
| pabelanger | AJaeger: sure, I can dive into that this afternoon | 11:33 |
| *** dalgaaf has joined #openstack-infra | 11:33 | |
| *** yamamoto has quit IRC | 11:34 | |
| AJaeger | pabelanger: I made a online fix for https://review.openstack.org/#/c/293719/ (puppet ubuntu-trusty change) | 11:34 |
| openstackgerrit | Merged openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job https://review.openstack.org/293130 | 11:34 |
| yolanda | sure | 11:34 |
| AJaeger | yolanda: can you restart the proposal salve again, please? | 11:34 |
| yolanda | ok | 11:34 |
| Zara | anyone have time to look at https://review.openstack.org/#/c/293367/ (storyboard email address config)? it has some +1s but no +2 or +A yet; should be the last step. | 11:34 |
| yolanda | i have the window for it opened :) | 11:34 |
| yolanda | done | 11:34 |
| openstackgerrit | Merged openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos https://review.openstack.org/293775 | 11:35 |
| *** rhallisey has joined #openstack-infra | 11:35 | |
| AJaeger | pabelanger: https://review.openstack.org/#/c/293767/1 will conflict with 293719. Could you rebase it on top of each other, please? | 11:36 |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| AJaeger | iff we want to move forward today with 293719... | 11:37 |
| openstackgerrit | Merged openstack-infra/project-config: Create new networking-ovn-native-l3-nv check job https://review.openstack.org/289547 | 11:38 |
| openstackgerrit | Merged openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu https://review.openstack.org/293012 | 11:38 |
| *** kzaitsev_mb has joined #openstack-infra | 11:39 | |
| *** mrmartin has quit IRC | 11:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Add javascript jobs to fuel-ui project https://review.openstack.org/292760 | 11:40 |
| openstackgerrit | Merged openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible https://review.openstack.org/293540 | 11:41 |
| *** thorst has joined #openstack-infra | 11:42 | |
| *** yamamoto has joined #openstack-infra | 11:42 | |
| mattt | hi all, we have a gate job initiated by tox (see https://github.com/openstack/openstack-ansible-os_designate/blob/master/tox.ini#L119-L121), when i run this locally stuff installed by ansible happens /outside/ of tox's functional venv, but in the gate it installs everything inside the venv | 11:42 |
| mattt | i literally invoke 'tox -e functional' in my test env, should i be invoking tox in a different manner? | 11:42 |
| *** vgridnev has joined #openstack-infra | 11:43 | |
| *** thorst has quit IRC | 11:43 | |
| *** thorst has joined #openstack-infra | 11:43 | |
| *** thorst_ has joined #openstack-infra | 11:44 | |
| *** placko_ has quit IRC | 11:44 | |
| *** placko_ has joined #openstack-infra | 11:45 | |
| pabelanger | mattt: what version of tox do you have locally? | 11:47 |
| *** _skolekonov is now known as skolekonov | 11:48 | |
| *** chlong has joined #openstack-infra | 11:48 | |
| mattt | pabelanger: 1.6.0 | 11:48 |
| *** thorst has quit IRC | 11:48 | |
| *** openstackgerrit has quit IRC | 11:48 | |
| mattt | pabelanger: i installed that from apt, let me try from pip | 11:48 |
| *** openstackgerrit has joined #openstack-infra | 11:48 | |
| *** Daisy has joined #openstack-infra | 11:48 | |
| pabelanger | mattt: I also see testenv doing things in your homedir too, so that is an issue | 11:49 |
| mattt | pabelanger: what version of tox is in the infra images, do you know off hand ? | 11:49 |
| AJaeger | mattt: look at the log file, it shows all versions | 11:50 |
| pabelanger | Yup | 11:50 |
| pabelanger | 2.x for sure | 11:50 |
| *** zeih has joined #openstack-infra | 11:51 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 11:51 |
| *** mtanino has joined #openstack-infra | 11:52 | |
| AJaeger | smarcet: will https://review.openstack.org/#/c/293586/ fix the openstackid/translate.openstack.org problems? | 11:53 |
| *** Daisy has quit IRC | 11:54 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 11:54 |
| AJaeger | yolanda: do you still have the proposal restart window open and can restart it, please? Might have been too soon to restart it ;( | 11:55 |
| yolanda | restarted | 11:56 |
| mattt | pabelanger: tox upgraded in my test env still uses /usr/local/lib, rather than .tox/function/lib | 11:56 |
| AJaeger | thanks again, yolanda | 11:57 |
| yolanda | no problem, that's easy :) | 11:57 |
| * AJaeger will be back later | 11:58 | |
| AJaeger | yolanda: works now! 2nd job running ;) | 11:58 |
| yolanda | cool. I didn't know what reusable slave function worked in that way now | 11:58 |
| yolanda | in downstream we had a parameter_function: reusable-slave per job | 11:59 |
| doug-fish | smarcet: I'm still concerned about the translate.openstack.org problems. It's not clear to me that the fix https://review.openstack.org/#/c/293586/ is related to it (maybe it is?) | 12:01 |
| *** HeOS has quit IRC | 12:01 | |
| doug-fish | is the fix related, or are you still working on translate.openstack.org separately? | 12:01 |
| pabelanger | okay, pushing 293719 now. | 12:01 |
| pabelanger | will monitor JJB | 12:01 |
| pabelanger | mattt: sorry, I don't have time to help debug this morning. Hopefully somebody else will be able to lend a hand | 12:02 |
| mattt | pabelanger: np, thanks anyway! | 12:02 |
| yolanda | ah, pabelanger has been the brave one to approve puppet moves to ubuntu-trusty :) | 12:03 |
| *** HeOS has joined #openstack-infra | 12:05 | |
| pabelanger | yolanda: Ya, I looked it over last night. | 12:05 |
| pabelanger | now to have JJB work :) | 12:06 |
| yolanda | pabelanger, care approving https://review.openstack.org/292792 ? to fix a devstack-gate bug | 12:06 |
| yolanda | thx | 12:11 |
| *** yamamoto has quit IRC | 12:14 | |
| *** weshay has joined #openstack-infra | 12:14 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 12:14 |
| pabelanger | yolanda: AJaeger: fungi lets reduce devstack-trusty to 1 for 293506 | 12:16 |
| pabelanger | so we can reduce the existing devstack-trusty nodes | 12:16 |
| nibalizer | fungi: okay, running the thing | 12:17 |
| *** thorst has joined #openstack-infra | 12:17 | |
| *** baoli has joined #openstack-infra | 12:18 | |
| *** _amrith_ is now known as amrith | 12:18 | |
| *** gordc has joined #openstack-infra | 12:18 | |
| openstackgerrit | Merged openstack-infra/project-config: The puppet-cookiecutter need gems to works. https://review.openstack.org/292933 | 12:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty https://review.openstack.org/293506 | 12:18 |
| *** edmondsw has joined #openstack-infra | 12:18 | |
| pabelanger | yolanda: AJaeger: fungi: these land that first, then move devstack-trusty to 0 after we land puppet JJB | 12:19 |
| doug-fish | Maybe smarcet is off now? Is anyone else looking at the translate.openstack.org problem? Many (all?) translators have been unable to translate for the last 12 hours or so. | 12:19 |
| pabelanger | as we have 159 ready devstack-trusty images | 12:19 |
| pabelanger | doug-fish: sadly, I don't know the status atm | 12:19 |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** armax has quit IRC | 12:20 | |
| *** hashar is now known as hasharAway | 12:21 | |
| *** aysyd has joined #openstack-infra | 12:21 | |
| *** baoli_ has joined #openstack-infra | 12:21 | |
| *** thorst_ has quit IRC | 12:21 | |
| *** jpr has joined #openstack-infra | 12:22 | |
| doug-fish | pabelanger: understood - I guess I'm looking for at least some acknowledgment that it's recognized as a significant problem and is a priority for somebody. | 12:22 |
| *** BobBall_ has joined #openstack-infra | 12:23 | |
| ianychoi | doug-fish, I have tried to find why openstackid is not working, but I am not an infra guy, so it is hard to identify this bug.. | 12:23 |
| *** BobBall_ is now known as Guest47198 | 12:23 | |
| ianychoi | For openstack-id, smarcet is the person who commits into openstackid, see: https://review.openstack.org/#/q/project:openstack-infra/openstackid | 12:23 |
| ianychoi | I think he would be the best person to solve this important problem. | 12:24 |
| *** baoli has quit IRC | 12:24 | |
| pabelanger | doug-fish: I can follow up, like I'm sure you will, but openstackid.org is a mystery to me. Hopefully another infra-root knows a little more | 12:25 |
| doug-fish | ianychoi: good to know - thanks for the background! Are other systems unavailable as well? I was only aware of this as a translate.openstack.org problem | 12:25 |
| *** dims has quit IRC | 12:25 | |
| *** BobBall has quit IRC | 12:25 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| ianychoi | translate-dev.o.o and refstack.o.o also do not work now. | 12:26 |
| doug-fish | I see. | 12:26 |
| *** vgridnev has quit IRC | 12:26 | |
| nibalizer | pabelanger: i don't kno wmuch either | 12:27 |
| *** tzn has quit IRC | 12:27 | |
| *** automagically_ has joined #openstack-infra | 12:30 | |
| pabelanger | yolanda: looks like 293719 will need some more work, failing gate now. So, removing +A atm, as I might have to step away | 12:30 |
| yolanda | chekcing | 12:30 |
| *** automagically has joined #openstack-infra | 12:31 | |
| pabelanger | nibalizer: do you have a moment to approve 293506? we're reducing devstack-trusty nodes and increasing ubuntu-trusty | 12:31 |
| *** vgridnev has joined #openstack-infra | 12:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Use pip installed dib for functional tests https://review.openstack.org/293854 | 12:33 |
| *** automagically_ has quit IRC | 12:33 | |
| *** asettle has joined #openstack-infra | 12:34 | |
| *** electrofelix has joined #openstack-infra | 12:34 | |
| nibalizer | sue | 12:35 |
| nibalizer | sure | 12:35 |
| *** dguitarbite has joined #openstack-infra | 12:37 | |
| *** automagically has left #openstack-infra | 12:37 | |
| *** Keedya has joined #openstack-infra | 12:37 | |
| mnaser | unrelated but i feel like infra team might have a thought here, anyone know of a tool which can show iops per process in linux? iotop does bandwidth per process which isn't answering our specific question | 12:38 |
| odyssey4me | clarkb mordred Based on https://review.openstack.org/293155 it looks like we could start to add additional mirrors into the apt mirror? Would it be acceptable, for instance, to add a MariaDB mirror? | 12:39 |
| *** kgiusti has joined #openstack-infra | 12:39 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| evgenyl | Hi, kindly ask to consider reviewing/merging patch on repo creation https://review.openstack.org/#/c/292765/ , thanks. | 12:40 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/system-config: Remove http.maxWait hack as fixed in Gerrit 2.5 https://review.openstack.org/207941 | 12:42 |
| *** piet has joined #openstack-infra | 12:42 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Refactoring https://review.openstack.org/293992 | 12:42 |
| *** hasharAway is now known as hashar | 12:42 | |
| *** armax has joined #openstack-infra | 12:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty https://review.openstack.org/293506 | 12:44 |
| *** hashar has quit IRC | 12:45 | |
| *** Keedya has quit IRC | 12:45 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Reduce exceptions noise https://review.openstack.org/293994 | 12:46 |
| *** eharney has joined #openstack-infra | 12:47 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Reduce exceptions noise https://review.openstack.org/293994 | 12:47 |
| *** hashar has joined #openstack-infra | 12:47 | |
| electrofelix | regarding irc logs of openstack related channels, is that reserved for just big tent projects, or is it something that any of the projects under the openstack umbrella can avail of? | 12:47 |
| *** salv-orlando has joined #openstack-infra | 12:48 | |
| *** yamamoto has quit IRC | 12:48 | |
| *** Daisy has joined #openstack-infra | 12:50 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13 https://review.openstack.org/293998 | 12:50 |
| pabelanger | okay, stepping away for a few hours. Running over the to USA! | 12:51 |
| ianychoi | smarcet, fungi, could you review https://review.openstack.org/293998 ? It is so important problem I think.. Please review it and if it is not the solution, would you find a solution for openstackid problems? | 12:51 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13 https://review.openstack.org/293998 | 12:52 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13 https://review.openstack.org/293998 | 12:53 |
| Daisy | ianychoi: Thank you for working on the login problem. | 12:53 |
| *** esikachev has joined #openstack-infra | 12:54 | |
| pabelanger | ianychoi: +2 to rollback, if openstackid stopped working after the patch I approved landed | 12:54 |
| pabelanger | assuming rollbacks work | 12:55 |
| ianychoi | pabelanger, thanks and really sorry for your review removement. | 12:55 |
| *** amrith is now known as _amrith_ | 12:55 | |
| ianychoi | I tried to add an reviewer: smarcet but mis-removed you.. | 12:56 |
| Daisy | smarcet, fungi: please help to fix the login problem of translation website as soon as possible. If the patch ianychoi submitted can work as a temp solution, please approve it. We need the translation website workable. It's a busy time for translators. And they have been stand by for a whole day. | 12:56 |
| smarcet | hey Daisy | 12:56 |
| smarcet | sorry which is the issue? | 12:56 |
| ianychoi | smarcet, would you see https://review.openstack.org/#/c/293998 ? | 12:57 |
| Daisy | We cannot login to http://www.openstackorg with Openstack ID now. | 12:57 |
| *** amotoki has joined #openstack-infra | 12:57 | |
| pabelanger | infra-root, if you have a moment, could use some feedback on our ongoing openid issue ^ | 12:58 |
| *** yamamoto has joined #openstack-infra | 12:58 | |
| *** salv-orlando has quit IRC | 12:58 | |
| smarcet | Daisy : mmm https://www.openstack.org/ login is working for me | 12:58 |
| Zara | someone has made a story here if it's useful to update things there: https://storyboard.openstack.org/#!/story/2000513 | 12:59 |
| Daisy | smarcet: sorry wrong website | 12:59 |
| Daisy | smarcet: try this: http://translate.openstack.org | 12:59 |
| Zara | (either way, when things are fixed, please someone let the author know.) | 13:00 |
| smarcet | Daisy: should i ask are u using DH association type? | 13:00 |
| smarcet | bc i am seeing a lot of errors related to that | 13:01 |
| smarcet | 'Zend\Crypt\Exception\InvalidArgumentException' with message 'Invalid | 13:01 |
| smarcet | parameter; not a positive natural number greater than 1' in | 13:01 |
| smarcet | Stack trace: #0 | 13:01 |
| smarcet | Zend\Crypt\PublicKey\DiffieHellman->setGenerator('1') #1 | 13:01 |
| Daisy | smarcet: I don't know, sorry. pleia2 should know the answer. | 13:01 |
| smarcet | before rollback openstackid | 13:02 |
| smarcet | please let me know that | 13:02 |
| smarcet | bc i think that request are passing wroing parameters | 13:02 |
| Daisy | smarcet: I have to join i18n meeting now. talk to you later. | 13:03 |
| smarcet | we are not using DH association on www.openstack.org | 13:03 |
| smarcet | k | 13:03 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Improve TripleO testing coverage for Puppet OpenStack CI https://review.openstack.org/294009 | 13:03 |
| *** katomo has joined #openstack-infra | 13:03 | |
| openstackgerrit | Merged openstack-infra/openstackid: Refactoring https://review.openstack.org/293992 | 13:04 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Improve TripleO testing coverage for Puppet OpenStack CI https://review.openstack.org/294009 | 13:05 |
| smarcet | pabelanger:ianychoi: need to know before rollback which is the openid association type that site is using | 13:05 |
| ianychoi | smarcet, it was working so well yesterday. | 13:06 |
| ianychoi | I tried to identify what have occurred for one day, and just found that openstackid version was changed.. | 13:07 |
| *** mtanino has quit IRC | 13:07 | |
| smarcet | well thing is that i need to know which kind of association are u using | 13:07 |
| smarcet | to identify the problem | 13:07 |
| smarcet | www.openstack.org site is working fine so far, but no use DH assoc | 13:08 |
| *** asettle has quit IRC | 13:08 | |
| pabelanger | smarcet: I'm leaving for a few hours, but will be back. So another root will take over the patch. We should likely have some sort of recap once we figure out the issue so we don't hit the problem again. | 13:08 |
| ianychoi | Hmm.. refstack.openstack.org is not also working. | 13:09 |
| ianychoi | https://github.com/openstack/refstack/blob/e35a0e3429bbd9311894608d10c0698becbd634d/refstack/api/controllers/auth.py#L2 | 13:09 |
| ianychoi | Would something with old openstackid access method would generate some problems? | 13:09 |
| smarcet | ianychoi: thing is that 1.0.14 did touch anything related the openid flow | 13:10 |
| smarcet | only related to oauth2 | 13:10 |
| *** markvoelker has joined #openstack-infra | 13:11 | |
| AJaeger | pabelanger: I'll rebase 293719 now... | 13:11 |
| * Zara reads logs, sees openid story was generated thanks to suggestions in this channel, wanders off again :) | 13:12 | |
| pabelanger | AJaeger: great. | 13:12 |
| *** jpr has quit IRC | 13:13 | |
| *** asettle has joined #openstack-infra | 13:13 | |
| pabelanger | okay, finally stepping away. devstack-trusty is at min-ready: 1 now. but we have 148 ready ATM. That will go down until puppet JJB change changes | 13:13 |
| pabelanger | will back in a few hours | 13:13 |
| *** asettle has quit IRC | 13:13 | |
| *** salv-orlando has joined #openstack-infra | 13:15 | |
| *** ajmiller has joined #openstack-infra | 13:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 13:17 |
| *** pradk has joined #openstack-infra | 13:18 | |
| *** akshai has quit IRC | 13:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 13:19 |
| *** amitgandhinz has joined #openstack-infra | 13:19 | |
| *** thiagop_ has joined #openstack-infra | 13:19 | |
| *** bdemers has quit IRC | 13:19 | |
| *** placko_ has quit IRC | 13:21 | |
| AJaeger | pabelanger, clarkb, yolanda, this passes local testing now. I rebased first and then fixed the issue that was introduced due to other change getting approved ^ | 13:21 |
| *** placko_ has joined #openstack-infra | 13:22 | |
| openstackgerrit | Sam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces https://review.openstack.org/293648 | 13:22 |
| *** mrmartin has joined #openstack-infra | 13:23 | |
| *** thiagop has quit IRC | 13:23 | |
| smarcet | yolanda: fungi: could i get access to openstackid.org server | 13:23 |
| smarcet | ? | 13:23 |
| smarcet | so i could take a look | 13:23 |
| smarcet | https://review.openstack.org/#/c/293491/ | 13:23 |
| *** thiagop_ is now known as thiagop | 13:24 | |
| *** mriedem has joined #openstack-infra | 13:24 | |
| *** asettle has joined #openstack-infra | 13:25 | |
| AJaeger | ARgh, proposal slave is down again ;( yolando, could you try again and check the logs, please? | 13:27 |
| *** links has quit IRC | 13:28 | |
| *** yamamoto has quit IRC | 13:28 | |
| *** oanson has quit IRC | 13:29 | |
| *** asettle has quit IRC | 13:30 | |
| *** _amrith_ is now known as amrith | 13:31 | |
| SamYaple | mordred: w00t more crazy glance things hoooo | 13:31 |
| SamYaple | should fix the OVH upload issues yes clarkb ? | 13:32 |
| *** akshai has joined #openstack-infra | 13:32 | |
| *** moravec has quit IRC | 13:32 | |
| *** yamamoto has joined #openstack-infra | 13:34 | |
| amrith | hi, how does one get a custom dashboard installed for a project. I'm looking to add some dashboards and would like them to, maybe, show up in https://review.openstack.org/#/admin/projects/openstack/trove,dashboards. I'll likely create them with gerrit-dash-creator and would like the link that it creates to show up here ... is that possible? | 13:35 |
| *** krtaylor has quit IRC | 13:36 | |
| *** _ari_ has joined #openstack-infra | 13:36 | |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| AJaeger | amrith: best talk with jeblair and drag rossella_s in - this would fullfill a request by neutron as well. But it's not clear yet how to do. jeblair gave some suggestions last Friday (see eavesdrop.o.o) but this needs more work | 13:37 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: WIP: Add a job to build packages https://review.openstack.org/294022 | 13:38 |
| *** regXboi has joined #openstack-infra | 13:38 | |
| *** hashar has quit IRC | 13:38 | |
| rossella_s | amrith, I've create a script for neutron...https://github.com/openstack/neutron/blob/master/tools/milestone-review-dash.py | 13:38 |
| rossella_s | amrith, I am looking for a way to have custom dashboard too...so if you find out how to do that, let me know :) | 13:39 |
| *** hamzy has quit IRC | 13:40 | |
| rossella_s | AJaeger, thanks a lot for pinging me! | 13:40 |
| *** yamamoto has quit IRC | 13:40 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| *** ddieterly has joined #openstack-infra | 13:40 | |
| *** fhubik has quit IRC | 13:40 | |
| *** jamesmcarthur has joined #openstack-infra | 13:40 | |
| zigo | AJaeger: Am I right that the Debian image was successfully built? | 13:41 |
| *** shardy has joined #openstack-infra | 13:41 | |
| amrith | AJaeger, did you say jeblair gave you directions on this channel? | 13:41 |
| amrith | rossella_s, thanks. | 13:41 |
| AJaeger | zigo: no idea ;( I don't have access to these systems | 13:41 |
| *** ajmiller has quit IRC | 13:42 | |
| AJaeger | amrith: yes, he did - read logs from last friday on eavesdrop | 13:42 |
| SamYaple | zigo: w00t debian | 13:42 |
| *** katomo has left #openstack-infra | 13:42 | |
| amrith | AJaeger, thanks | 13:42 |
| *** oanson has joined #openstack-infra | 13:42 | |
| amrith | I'm trying to do one other thing. Find a way to tag reviews into groups. like Mitaka-3, or Mitaka-RC1. Right now the best I'm able to do is star the reviews and then share a link that includes starredby:amrith | 13:42 |
| amrith | this seems lame. | 13:42 |
| amrith | I don't know how to tag reviews. | 13:42 |
| amrith | if rossella_s or AJaeger know how to do that, please share ... | 13:42 |
| zigo | AJaeger: I'm double guessing that it did work, because someone pushed the .qcow2 image at http://nodepool.openstack.org/176625.qcow2 as I asked for (so I can inspect the image and design my script). | 13:43 |
| zigo | AJaeger: Is it correct to say that the logs at http://nodepool.openstack.org/ are combined logs, and that http://nodepool.openstack.org/dib.debian-jessie.log may contain multiple build attempts before logrotate kicks in? | 13:43 |
| AJaeger | zigo: Sorry, I cannot really help with image building, please wait for other infra folks and discuss with them | 13:44 |
| AJaeger | amrith: one way: Use the same topic ;) | 13:44 |
| zigo | AJaeger: Thanks, will do. | 13:44 |
| rossella_s | amrith, this is the irc logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-11.log.html#t2016-03-11T14:46:05 | 13:44 |
| *** moravec has joined #openstack-infra | 13:45 | |
| *** piet has quit IRC | 13:45 | |
| amrith | thanks, I was having a hard time finding it ... | 13:45 |
| amrith | AJaeger, use the same topic is an interesting approach | 13:45 |
| jamesmcarthur | fungi: groups is working fine as is www.o.o | 13:45 |
| amrith | I guess I can go and edit the topics | 13:45 |
| amrith | and include something common. | 13:46 |
| amrith | still seems hokey ;) | 13:46 |
| jamesmcarthur | talking with smarcet: this morning and it seems like we’ve got two separate errors | 13:46 |
| openstackgerrit | Rob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack https://review.openstack.org/293090 | 13:46 |
| jamesmcarthur | refstack atually gets you to the point of logging in, but then throws a 500 | 13:46 |
| *** hashar has joined #openstack-infra | 13:46 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:47 | |
| *** nwkarsten has joined #openstack-infra | 13:47 | |
| jamesmcarthur | These errors started rolling in before the latest patch was approved for OpenStackID. This is just the first time we’ve been able to see the logs. | 13:47 |
| *** krtaylor has joined #openstack-infra | 13:48 | |
| jamesmcarthur | fungi: Try logging into refstack and you get teh proper openid assertion | 13:48 |
| jamesmcarthur | https://refstack.openstack.org/api/v1/auth/signin_return?csrf_token=vevvloqogsoubrugirqjzrgrgwgwnj&openid.assoc_handle=oJ~iKtv2_k3Qi3f~KpRu2nO7ThTVt6DJ&openid.claimed_id=https%3A%2F%2Fopenstackid.org%2Fjimmy.mcarthur&openid.identity=https%3A%2F%2Fopenstackid.org%2Fjimmy.mcarthur&openid.mode=id_res&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.sreg=http%3A%2F%2Fopenid.net%2Fextensions%2Fsreg%2 | 13:48 |
| jamesmcarthur | F1.1&openid.op_endpoint=https%3A%2F%2Fopenstackid.org%2Faccounts%2Fopenid2&openid.realm=https%3A%2F%2Frefstack.openstack.org%2Fapi&openid.response_nonce=2016-03-17T13%3A44%3A09ZrwAFMhkd381dGRQ2MfiI2SjUXxtZFwbs&openid.return_to=https%3A%2F%2Frefstack.openstack.org%2Fapi%2Fv1%2Fauth%2Fsignin_return%3Fcsrf_token%3Dvevvloqogsoubrugirqjzrgrgwgwnj&openid.sig=5NZtcKQ8ZoFZCVlBl2QXUNfk%2FmWYtQfgVVfjhF28wY4%3D&openid.s | 13:48 |
| jamesmcarthur | gned=op_endpoint%2Crealm%2Creturn_to%2Cresponse_nonce%2Cassoc_handle%2Cclaimed_id%2Cidentity%2Csreg.email%2Csreg.fullname%2Csigned&openid.sreg.email=jimmy%40openstack.org&openid.sreg.fullname=Jimmy+McArthur | 13:48 |
| AJaeger | jamesmcarthur: what about translate.openstack.org? That is release critical | 13:48 |
| jamesmcarthur | but refstack throws this error - {"code": 500, "title": "Internal Server Error"} | 13:48 |
| smarcet | Ajaeger: seems that there is some problem on DH that they are seing | 13:49 |
| smarcet | sending | 13:49 |
| jamesmcarthur | Ajaeger: translate is throwing a different error | 13:50 |
| jamesmcarthur | [2016-03-17 13:49:21] dev.ERROR: exception | 13:50 |
| jamesmcarthur | 'Zend\Crypt\Exception\InvalidArgumentException' with message 'Invalid | 13:50 |
| jamesmcarthur | parameter; not a positive natural number greater than 1' in | 13:50 |
| jamesmcarthur | Stack trace: #0 | 13:50 |
| smarcet | they seems to sending a bad crypto param | 13:50 |
| AJaeger | jamesmcarthur: plesae use paste.openstack.org for such long pastes | 13:50 |
| smarcet | we expect greater or equal to 2 | 13:50 |
| smarcet | they send 1 | 13:50 |
| AJaeger | smarcet, jamesmcarthur can we go back to a working version, debug further and then continue? | 13:51 |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| AJaeger | The translators need access to translate.openstack.org to translate the Mitaka release | 13:52 |
| smarcet | AJaeger: would be helpfull if we got some examples of the openid requests that translate is doing | 13:52 |
| *** ifarkas has quit IRC | 13:52 | |
| AJaeger | smarcet: I'm not an infra-root and can't help on that front | 13:52 |
| *** tzn has joined #openstack-infra | 13:53 | |
| *** ddieterly is now known as ddieterly[away] | 13:53 | |
| smarcet | me neither ;) | 13:53 |
| *** dims has joined #openstack-infra | 13:53 | |
| *** piet has joined #openstack-infra | 13:53 | |
| jamesmcarthur | Ajaegar: we are in a precarious position here b/c we are right in the middle of testing iOS apps. If we roll back, they break and we can’t submit app for approval in time for summit. | 13:53 |
| smarcet | smarcet: IHMO, we started to see those errors a long before than 1.0.14 was released | 13:53 |
| *** yamamoto has quit IRC | 13:53 | |
| yolanda | smarcet sorry i was out | 13:54 |
| *** xyang1 has joined #openstack-infra | 13:54 | |
| jamesmcarthur | smarcet introduced a patch to get root access to the OpenStackID server. I think we could get this straightened out fairly quickly if we could just get access to this server. | 13:54 |
| jamesmcarthur | if we roll back, we’re not going to be able to figure out what the error was | 13:54 |
| yolanda | smarcet, i +2 your change, it needs another infra-root there | 13:55 |
| jamesmcarthur | yolanda: thank you! | 13:55 |
| *** sshnaidm has quit IRC | 13:55 | |
| smarcet | yolandaL thx u! | 13:55 |
| smcginnis | Gerrit permissions question - I wasn't in cinder-milestone so I didn't have +2/+A on the new stable/mitaka branch. | 13:56 |
| *** bdemers has joined #openstack-infra | 13:56 | |
| smcginnis | I've been added now, but after logging out and back in I still only have -1/+1. | 13:56 |
| smcginnis | Does it take a bit for those permissions to propogate? | 13:56 |
| *** mtanino has joined #openstack-infra | 13:57 | |
| amrith | AJaeger, rossella_s ... this may help. http://chris.wang/posts/2016/01/04/gerrit-dashboards | 13:58 |
| jamesmcarthur | Ajaegar: I think where we’re also stuck is there were 0 changes on openid b/w releases. It’s not impossible that something in this release broke, but the fact that these errors are coming from openid, not oAuth2 doesn’t poitn to that. Hopefully if smarcet can get another +2, we can see what’s happening pretty quickly. | 13:58 |
| Daisy | smarcet: I'm back. | 13:59 |
| AJaeger | amrith: the question is how to *push* it there. neutron wants to update the dashboard daily | 13:59 |
| Daisy | smarcet: any progress with the login issue ? | 13:59 |
| *** e0ne has quit IRC | 13:59 | |
| amrith | AJaeger, I guess one finds how to do it once, then use the formula found on any bottle of shampoo ... | 13:59 |
| amrith | ;) | 13:59 |
| amrith | I'm still trying to figure out how to do it the first time. | 14:00 |
| *** e0ne has joined #openstack-infra | 14:00 | |
| amrith | right now, having trouble understanding how to get to /refs/metadata/... | 14:00 |
| rossella_s | amrith, that's in the server where gerrit runs I think | 14:01 |
| *** jamesmcarthur has quit IRC | 14:01 | |
| amrith | and this airline wifi is flakey ;( | 14:01 |
| smarcet | Daisy: | 14:01 |
| smarcet | i dont have access to server | 14:01 |
| *** jamesmcarthur has joined #openstack-infra | 14:01 | |
| smarcet | more over i dont have any sample requests from translate | 14:01 |
| amrith | well rossella_s the things in that chris wang blog post actually do get me to the config section of the project. | 14:01 |
| amrith | i.e. if'm doing this on a clone of the trove project | 14:01 |
| smarcet | i am suspect that translate is sending openid.dh_gen = 1 | 14:01 |
| smarcet | and that is the problem | 14:01 |
| amrith | and when I git fetch ... I end up seeing the project config | 14:02 |
| amrith | I guess one has to push that | 14:02 |
| amrith | and for that one would probably need to be an admin of some kind | 14:02 |
| rossella_s | amrith, no idea how infra manages that, sorry | 14:02 |
| fungi | 500 lines of scrollback. should catch up shortly | 14:02 |
| *** ddieterly[away] is now known as ddieterly | 14:03 | |
| amrith | well, a step in the right direction takes you a step closer to the end point. | 14:03 |
| amrith | thanks rossella_s AJaeger | 14:03 |
| rossella_s | amrith, you should have a loot at jeepyb | 14:03 |
| *** ifarkas has joined #openstack-infra | 14:03 | |
| AJaeger | fungi, basically two problems right now: 1) openstackid and translate.openstack.org not working yet and 2) proposal slave goes offline. We thought it was fixed but seems not permanent ;( | 14:03 |
| amrith | got it, http://docs.openstack.org/infra/system-config/jeepyb.html | 14:04 |
| Daisy | smarcet: cannot we just rollback ? I don't think our translators can wait. We are in busy time now. | 14:04 |
| rossella_s | amrith, didn't have much time till now but that's my next step to try to understand how to make the dashboard available | 14:04 |
| * AJaeger has to step out | 14:04 | |
| rossella_s | amrith, I found this old review, not sure if we can still use something from it https://review.openstack.org/#/c/94260/ | 14:04 |
| fungi | AJaeger: proposal.slave.openstack.org is indeed offline. it looks like the last job it ran was django_openstack_auth-propose-translation-update-mitaka and that offlined it, so i bet we're missing an adjustment to the zuul parameter script to recognize the mitaka translations as a job name pattern to not offline after | 14:04 |
| fungi | yolanda: ^ | 14:04 |
| fungi | oh, looks like there's a fix | 14:05 |
| rcarrillocruz | yeah | 14:05 |
| yolanda | again? | 14:05 |
| fungi | which is merged, but... did anyone re-online the worker? it's still offline | 14:05 |
| rcarrillocruz | is it down again? cos I pushed a change earlier that got merged | 14:06 |
| yolanda | i put online a bunch of times | 14:06 |
| rcarrillocruz | :S | 14:06 |
| yolanda | going to do it again | 14:06 |
| *** nwkarsten has quit IRC | 14:06 | |
| yolanda | so maybe i didn't replicate to jenkins masters | 14:06 |
| yolanda | stable-mitaka are turning the node off on each run | 14:07 |
| smarcet | Daisy: i dont have the permissions to do that | 14:07 |
| *** nwkarsten has joined #openstack-infra | 14:07 | |
| *** infra-red has joined #openstack-infra | 14:08 | |
| *** infra-red has quit IRC | 14:08 | |
| jamesmcarthur | Daisy: based on the changes that went through and the errors we’re seeing from translate it seems we might be dealign with another issue. | 14:08 |
| *** elmiko has joined #openstack-infra | 14:08 | |
| openstackgerrit | Michael McCune proposed openstack-infra/irc-meetings: correct data processing meeting times https://review.openstack.org/294042 | 14:08 |
| jamesmcarthur | We are all under major crunch time. If smarcet can get root accesss to the server, we can help troubleshoot pretty quickly. | 14:09 |
| Daisy | smarcet, fungi, AJaeger : As to the translation website login issue, I think the issue can be resolved by pleia, Zanata developers, and smarcet together. But because of the time difference, Zanata developers are not available now. I don't have the right to access that server too. But it's the most busy time in a release for translators. We have only 1 week left. We need to have the server... | 14:09 |
| Daisy | ...available ASAP. Please, find a way to bring the server back to translators. please. | 14:10 |
| *** piet has quit IRC | 14:10 | |
| ianychoi | Please.. +1 for Daisy's request | 14:10 |
| *** nwkarsten has quit IRC | 14:11 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Added mising link to config file https://review.openstack.org/293586 | 14:14 |
| *** moravec has quit IRC | 14:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 14:15 |
| *** jaypipes has quit IRC | 14:15 | |
| *** piet has joined #openstack-infra | 14:15 | |
| *** placko_ has quit IRC | 14:16 | |
| fungi | smarcet: jamesmcarthur: i believe pleia2 and StevenK worked on the openid configuration for zanata on translate.openstack.org but they're in singapore and australia right now i think. we can collectively start reverse-engineering, but it looks like we're just passing an openid url to a zanata puppet module red hat provided. i'm digging up the details now | 14:16 |
| *** placko_ has joined #openstack-infra | 14:16 | |
| *** tonytan4ever has joined #openstack-infra | 14:16 | |
| jamesmcarthur | fungi: would you be able to +2 smarcet’s request for root access so he can review the request in the logs? | 14:17 |
| fungi | jamesmcarthur: which one? i already approved one a little bit ago | 14:17 |
| *** esikachev has quit IRC | 14:18 | |
| smarcet | https://review.openstack.org/#/c/293491/ | 14:18 |
| jamesmcarthur | ah - that was it :) | 14:18 |
| jamesmcarthur | oh, not that was not it ;) | 14:18 |
| jamesmcarthur | it’s the one Yolanda +2’d | 14:18 |
| *** ajmiller has joined #openstack-infra | 14:19 | |
| fungi | oh, the one where we're giving up on separation of concerns | 14:19 |
| fungi | anyway, let's go ahead and roll back. if it turns out that our need for a stable openid provider for our community systems is incompatible with the development model for the marketing website then we should probably plan to fork the codebase temporarily and stand up separate servers while we work out whether there's a simpler openid implementation we can migrate to | 14:19 |
| *** dims has quit IRC | 14:19 | |
| *** amrith is now known as _amrith_ | 14:19 | |
| Daisy | +1 to fungi . Thank you. | 14:20 |
| fungi | it doesn't sound like we can take the time for a lengthy debate with the root team right now about whether we're comfortable giving foundation contractors root accounts on the authentication system which was intended to serve as the security linchpin of all our production systems | 14:20 |
| ianychoi | +1 thanks a lot, fungi | 14:21 |
| *** jaypipes has joined #openstack-infra | 14:22 | |
| *** jamesmcarthur_ has joined #openstack-infra | 14:22 | |
| *** rbrndt has joined #openstack-infra | 14:22 | |
| *** dims has joined #openstack-infra | 14:22 | |
| fungi | separately, we can dig into what refstack, keystone and zanata are collectively doing which is incorrect in light of the recent changes to openstackid and try to get their respective upstreams to address those issues | 14:23 |
| jamesmcarthur_ | fungi: “oh, the one where we're giving up on separation of concerns” - I’m not sure I understand that. It’s pretty valuable for a dev team to have access to logs. | 14:23 |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| fungi | jamesmcarthur_: we set up a complete copy of the platform for that exact purpose | 14:23 |
| *** jamesmcarthur has quit IRC | 14:23 | |
| *** jamesmcarthur_ is now known as jamesmcarthur | 14:23 | |
| AJaeger | jamesmcarthur_: access to logs != root access | 14:23 |
| *** Guest47198 is now known as BobBall | 14:23 | |
| jamesmcarthur | fungi: but real traffic doesn’t hit the copy of the platform | 14:23 |
| fungi | there's a translate-dev.openstack.org which is supposed to be authenticating against openstackid-dev.openstack.org | 14:24 |
| AJaeger | jamesmcarthur: we have an translate-dev as well | 14:24 |
| AJaeger | fungi, so going forward with https://review.openstack.org/#/c/293998 by ianychoi ? | 14:24 |
| fungi | the intent was to use that to be able to test new changes to openstackid.org before updating it, so we could make sure we don't break the translation site authenticationm | 14:24 |
| *** dims has quit IRC | 14:25 | |
| fungi | also if you need logs data from openstackid.org i'm happy to provide it to you directly | 14:25 |
| jamesmcarthur | fungi: this is the first time we’ve heard of translate-dev or that translate is using openstackID. I’m glad they are :) but in the meantime, rolling back is going to break active fixes in our own development | 14:25 |
| *** yamamoto has quit IRC | 14:26 | |
| jamesmcarthur | fungi: so smarcet won’t get approved for root access to that server? | 14:26 |
| smarcet | guys | 14:27 |
| ianychoi | Yes, and translate-dev.o.o is not also working now. Maybe Zanata developers and pleia2 would help for logs on translate-dev.o.o and the integration between translate-dev and openstackid-dev later (I am not sure whether translate-dev uses openstackid or openstackid-dev) | 14:27 |
| *** mrmartin has quit IRC | 14:27 | |
| smarcet | translate dev is poiting to https://openstackid.org/accounts/user/login o_0 | 14:27 |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| *** gongysh has joined #openstack-infra | 14:27 | |
| *** pcaruana has quit IRC | 14:28 | |
| fungi | ianychoi: pleia2 is travelling and i don't know whether she sees irc which is why we need to reverse engineer this | 14:28 |
| Daisy | agree, ianychoi . +1 | 14:28 |
| *** mrmartin has joined #openstack-infra | 14:28 | |
| fungi | jamesmcarthur: i'm not going to make an on the spot decision to grant root access to servers to people who aren't already heavily involved in maintaining any of our other systems, particularly when that's the authentication system for several important community (not foundation) platforms | 14:28 |
| jamesmcarthur | fungi: we’ve been asking for this for months, so it’s not something brand new | 14:29 |
| ianychoi | fungi, would you approve https://review.openstack.org/#/c/293998/ ? It is for rollback from openstackid 1.0.14 to 1.0.13, the previous version which had been used up to one day ago. | 14:29 |
| *** achanda has joined #openstack-infra | 14:29 | |
| *** tongli has joined #openstack-infra | 14:29 | |
| jamesmcarthur | if you want to wait, I understand, but it feels like we are constantly battling against this, when it’s a very reasonable request to give the most knowledgeable dev access to logs in real time | 14:29 |
| ianychoi | Up to one day ago, translate.o.o was working so well. | 14:30 |
| fungi | jamesmcarthur: and so far what opinion there has been from the group has been against that idea. having a sudden breakage of the platform doesn't instill confidence that also granting sysadmin access for that to the developer team is wise | 14:30 |
| *** nwkarsten has joined #openstack-infra | 14:30 | |
| ianychoi | It is different from root access or log access issue.. | 14:30 |
| smarcet | ianychoi: Daisy:fungi: translate-dev is poiting to openstackid production | 14:30 |
| smarcet | we are only asking root access for openstackid.org server | 14:30 |
| *** esikachev has joined #openstack-infra | 14:30 | |
| fungi | smarcet: yes, i believe that was from when it was being used to test a different zanata release, so it needs to be switched back to authenticating against openstackid-dev again | 14:31 |
| AJaeger | jamesmcarthur: so, what can we do to get you more involved so that you can learn about the complete setup and realizes how systems are connected and gain the trust by the root team to include you or smarcet? | 14:31 |
| *** nwkarste_ has joined #openstack-infra | 14:31 | |
| *** placko_ has quit IRC | 14:31 | |
| ogelbukh_ | please, review the project add patch: https://review.openstack.org/#/c/286137/ looks like all relevant signoffs are given | 14:31 |
| jamesmcarthur | Ajaegar: we are not asking for sysadmin access. We are asking for root access to a single server. | 14:31 |
| *** nwkarste_ has quit IRC | 14:32 | |
| smarcet | fungi: there is any ref stack dev instance? | 14:32 |
| *** placko_ has joined #openstack-infra | 14:32 | |
| *** _amrith_ is now known as amrith | 14:32 | |
| *** nwkarste_ has joined #openstack-infra | 14:32 | |
| jamesmcarthur | Ajaegar: as far as getting us more involved, it would defintiely be helpful if we knew of dependencies on openstackID. groups.o.o is obviouisly still working just fine as is www.o.o. | 14:32 |
| *** gongysh has quit IRC | 14:32 | |
| jamesmcarthur | Those are dependcies we check for when testing. | 14:32 |
| fungi | smarcet: i encouraged the refstack maintainers to put one together but there isn't a dev platform for it yet, no | 14:33 |
| jamesmcarthur | refstack and translate were unkown to us | 14:33 |
| *** hamzy has joined #openstack-infra | 14:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Switch translate-dev to openstackid-dev https://review.openstack.org/294060 | 14:33 |
| *** oanson has quit IRC | 14:33 | |
| AJaeger | fungi, let's fix the translate-dev system ^ | 14:33 |
| fungi | already approved | 14:33 |
| AJaeger | wow! | 14:33 |
| ianychoi | :) | 14:33 |
| jamesmcarthur | fungi: so it looks like there are lots of things that might have caused this - no dev server for refstack and translate-dev pointing to openstackid production. I think it’s misplaced to put this on our development. | 14:34 |
| Daisy | when will it take affect ? | 14:34 |
| Daisy | I mean, translate-dev.o.o ? | 14:34 |
| ianychoi | Daisy, not yet unfortunately.. | 14:34 |
| Daisy | I can help to test. | 14:34 |
| *** nwkarsten has quit IRC | 14:34 | |
| fungi | this should hopefully give us insight into what's wrong between the expectations of zanata and openstackid master while we have openstackid.org temporarily rolled back | 14:34 |
| *** tonytan4ever has quit IRC | 14:34 | |
| ianychoi | Wow, good to hear that | 14:34 |
| AJaeger | Daisy: translate-dev: After merge and next puppet run - expect an hour... | 14:34 |
| *** gongysh has joined #openstack-infra | 14:35 | |
| Daisy | ok. 1 hour. wait. | 14:35 |
| *** piet has quit IRC | 14:37 | |
| *** gongysh has quit IRC | 14:38 | |
| fungi | jamesmcarthur: yes, i'm just looking for an expedient means of identifying the problem for now without altering the workflow. i agree having translate-dev left authenticating against openstackid.org production was not great, but i also don't know that anyone bothered to try it since i agree the changelog for the new openstackid release didn't imply any behavior changes to the openid protocol handler | 14:38 |
| AJaeger | yolanda, fungi Is there a jenkins that is still taking down the proposal slave? It seems to be gone missing again ;( | 14:38 |
| jamesmcarthur | fungi: right - our concern is that this has been potentially going on longer than we thought | 14:38 |
| fungi | AJaeger: horizon-propose-translation-update-mitaka took it offline this time | 14:39 |
| *** gongysh has joined #openstack-infra | 14:39 | |
| jamesmcarthur | and again, translate wasn’t on our list of things to test with openstackID | 14:39 |
| *** tonytan4ever has joined #openstack-infra | 14:39 | |
| *** sdake has joined #openstack-infra | 14:39 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:40 | |
| AJaeger | fungi, so is the regex wrong? Or jenkins.openstack.org not updated? | 14:40 |
| *** achanda has quit IRC | 14:40 | |
| *** erlon has joined #openstack-infra | 14:40 | |
| fungi | jamesmcarthur: right, i think whatever's broken translate logins was in the last 24 hours because they've been using it heavily. i don't know the frequency with which refstack is used but would need to hear more from its users to know a rough timeframe for when the issue started. also they wrote their own openid consumer so i wouldn't be surprised if it's generating some error messages | 14:40 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:41 | |
| * AJaeger simplifies that regexes... | 14:41 | |
| fungi | AJaeger: i've not had time to review the regex update change so i don't know, but my guess would be that it's wrong | 14:41 |
| sdague | fungi: fyi, this has broken a bunch of folks - https://review.openstack.org/#/c/293942/ | 14:42 |
| sdague | or, the lack of that | 14:42 |
| *** Qiming_ has joined #openstack-infra | 14:42 | |
| *** kaisers_ has joined #openstack-infra | 14:42 | |
| openstackgerrit | Merged openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13 https://review.openstack.org/293998 | 14:42 |
| *** piet has joined #openstack-infra | 14:42 | |
| anteaya | fungi: fires today, trying to find a way to help, so far it all looks like root things | 14:43 |
| *** jaosorior is now known as jaosorior_away | 14:43 | |
| fungi | sdague: that's fine for now, though we should probably move that entire list into devstack's early bootstrap phase at some point | 14:43 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things https://review.openstack.org/294071 | 14:44 |
| clarkb | sdague: there is a different change to fix it in dib direvtly based on scrollback | 14:44 |
| clarkb | however this shpuld go in devstack... | 14:44 |
| AJaeger | anteaya: can you tell me what'S wrong with the regex in https://review.openstack.org/293923 ? | 14:44 |
| clarkb | you dont need uuidgen as a dep to install packages right? | 14:44 |
| fungi | ideally devstack should be able to work on as minimal a base system as possible and install all the things it knows it needs | 14:45 |
| anteaya | AJaeger: I can look, I don't profess that regexes are my strength | 14:45 |
| AJaeger | So, regex changes from (constraints-.*|updates?|update-liberty|plugins-list))$ to (constraints-.*|updates?|update-(liberty|mitaka)|plugins-list))$ - and does not catch update-mitaka ;( | 14:45 |
| * anteaya looks | 14:45 | |
| AJaeger | anteaya: see above ^ | 14:45 |
| *** flepied has quit IRC | 14:45 | |
| AJaeger | clarkb: I had to rebase and fix your devstack-trusty puppet patch, hope it's fine now... | 14:45 |
| AJaeger | fix=fix after other changes merged | 14:45 |
| *** Qiming has quit IRC | 14:46 | |
| sdague | fungi: maybe, I also think that the goal in dib was to build common images that work for things, and in current definitions is about making all tools have to assume different minimums than in the real world | 14:46 |
| *** jamesmcarthur_ has joined #openstack-infra | 14:46 | |
| sdague | clarkb: I don't have scrollback, so that's the best I got. People were still complaining about the breaks this morning. | 14:47 |
| *** jamesmcarthur has quit IRC | 14:47 | |
| *** jamesmcarthur_ is now known as jamesmcarthur | 14:47 | |
| fungi | smarcet: jamesmcarthur: so digging into the zanata puppet module, the openid url is being passed directly into zanata's xml config file: https://git.openstack.org/cgit/openstack-infra/puppet-zanata/tree/templates/standalone.xml.erb#n419 | 14:48 |
| *** Daisy_ has joined #openstack-infra | 14:49 | |
| * AJaeger just tested locally the regex from 293923 and it matches on 'horizon-propose-translation-update-mitaka' | 14:49 | |
| AJaeger | fungi, clarkb: What kind of regex bug was on the bare-precise, does this hit us here perhaps as well? | 14:49 |
| *** Sukhdev has joined #openstack-infra | 14:49 | |
| fungi | smarcet: jamesmcarthur: there's also a security-domain name="zanata.openid" block which is basically just turning on the openid support in zanata and providing the provider url, so i think any alterations in dh options are likely going to imply changes in the zanata codebase | 14:50 |
| *** gongysh has quit IRC | 14:50 | |
| *** yamahata has joined #openstack-infra | 14:50 | |
| fungi | red hat's devs working on zanata are in australia i think, so it will take a little time to pull them into the discussion | 14:50 |
| smarcet | ok | 14:51 |
| fungi | i'm going to start digging up the zanata source code, though java makes me go cross-eyed after not too long | 14:51 |
| Daisy | fungi: yes, there are in Australia. | 14:51 |
| *** tonytan4ever has quit IRC | 14:51 | |
| Daisy | s/there/they | 14:51 |
| smarcet | https://github.com/zanata/zanata-server/blob/master/zanata-war/src/main/java/org/zanata/security/OpenIdLoginModule.java | 14:51 |
| AJaeger | DAisy, can you send an email to Alex and co and ask them to join us here once they are awake, please? | 14:52 |
| Daisy | ok. I will do it now. | 14:52 |
| Daisy | I can send him short message. | 14:52 |
| clarkb | sdague: uuid-runtime doesnt appear to be a valid fedora package name, util-linux is where that is hiding? | 14:52 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce https://review.openstack.org/292792 | 14:52 |
| sdague | clarkb: yeh | 14:52 |
| AJaeger | thanks, Daisy | 14:52 |
| sdague | it's only an ubuntu issue | 14:52 |
| clarkb | yes but that package install is for everything | 14:53 |
| sdague | ok, so is the issue the meta definitions in dib don't map all the way through? | 14:53 |
| anteaya | AJaeger: I don't see anything wrong with the new regex, the one that has update-(liberty|mitaka) in it | 14:53 |
| *** Daisy_ has quit IRC | 14:53 | |
| anteaya | AJaeger: so perhaps it is as you say and there is a bug with the parser | 14:54 |
| clarkb | you didnt update the meta def I dont know that they ship a complete one for you | 14:54 |
| fungi | AJaeger: zuul's status page says "Last reconfigured: Thu Mar 17 2016 12:39:07 GMT+0000 (UTC)" so was that before the regex update merged? | 14:54 |
| clarkb | or ship one at all | 14:54 |
| AJaeger | fungi, that should be the time the regex update merged | 14:55 |
| *** piet has quit IRC | 14:56 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build packages https://review.openstack.org/294022 | 14:56 |
| fungi | yep, https://review.openstack.org/293923 merged at 10:59 utc so should be well before | 14:56 |
| *** piet has joined #openstack-infra | 14:56 | |
| *** Zhongjun_ has joined #openstack-infra | 14:56 | |
| *** bdemers has quit IRC | 14:57 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things https://review.openstack.org/294071 | 14:57 |
| *** tonytan4ever has joined #openstack-infra | 14:58 | |
| *** amitgandhinz has quit IRC | 14:58 | |
| fungi | AJaeger: it's possible these jobs were already in the gearman queue with the old parameter | 14:58 |
| *** amitgandhinz has joined #openstack-infra | 14:58 | |
| fungi | clarkb: ^ yeah? | 14:58 |
| *** nadya has quit IRC | 14:59 | |
| AJaeger | fungi, yes, they were | 14:59 |
| AJaeger | fungi, they were already in zuul since 6:00 UTC | 14:59 |
| fungi | parameters are going to be in the work request, i believe, so if they were queued in gearman before the reconfigure i think they retain their parameters | 14:59 |
| *** nadya has joined #openstack-infra | 14:59 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 15:00 |
| AJaeger | So, let's restart the workers once the queue is empty? | 15:00 |
| *** ifarkas_ has joined #openstack-infra | 15:00 | |
| fungi | which means our jenkins admins will probably just keep onlining the worker until those flush through, unless we want to restart zuul | 15:00 |
| AJaeger | There'S one more job with -mitaka in the queue and then we should be good | 15:00 |
| AJaeger | Thus one more restart I hope ;) | 15:00 |
| *** zz_dimtruck is now known as dimtruck | 15:01 | |
| *** nadya has quit IRC | 15:02 | |
| *** nadya has joined #openstack-infra | 15:02 | |
| clarkb | sdague: ya doesnt look like the pkg-map element has any default mappings. | 15:02 |
| *** ifarkas has quit IRC | 15:03 | |
| *** sshnaidm has joined #openstack-infra | 15:03 | |
| fungi | smarcet: it looks like most of the base openid options are being inherited from https://github.com/zanata/zanata-server/blob/master/zanata-war/src/main/java/org/zanata/security/openid/GenericOpenIdProvider.java | 15:03 |
| AJaeger | Time now for the final restart of the proposal slave, please | 15:03 |
| *** achanda has joined #openstack-infra | 15:03 | |
| clarkb | we just need to provide a pkg-map file and should be good | 15:03 |
| *** hashar is now known as hasharMeeting | 15:03 | |
| AJaeger | Daisy, ianychoi, doug-fish, fungi, smarcet, jamesmcarthur : I can access translate.openstack.org again! | 15:04 |
| Daisy | wow | 15:04 |
| Daisy | how did you make it happen ? | 15:05 |
| fungi | smarcet: i'm looking through some of the subclassing of that now to see what they do for some of the specific providers, as that mighth indicate overrides we'd be able to use | 15:05 |
| Daisy | AJaeger, I 'm in too. | 15:05 |
| zaro | sdague: here's the project for integrating ci test results to Gerrit https://gerrit.googlesource.com/plugins/verify-status/+/master/src/main/resources/Documentation | 15:06 |
| smarcet | Daisy: ok i will test dev version of openstackid with dev version of translate once its up | 15:06 |
| Daisy | AJaeger: I have sent email to Alex, cc you. I also leave him a short message. I think he will come here when he sees it. | 15:06 |
| doug-fish | AJaeger: that's great! | 15:06 |
| smarcet | fungi: i am still seeing the error of DH params | 15:06 |
| smarcet | seems that wasnt translate after all | 15:06 |
| ianychoi | Me also works translate.o.o well, thanks all! | 15:07 |
| zaro | sdague: only backend stuff so far, we are hoping to realize the displaying pretty soon. | 15:07 |
| smarcet | fungi: waiting to test on dev | 15:07 |
| ianychoi | Also, refstack works well | 15:07 |
| sdague | zaro: cool | 15:07 |
| fungi | nibalizer: did you have any luck with the owners.py run? instructions clear enough?need any help cross-checking the results? | 15:08 |
| Daisy | can I announce the server is back to i18n mailing list ? | 15:08 |
| Daisy | or you still want to do some more test ? | 15:08 |
| *** achanda has quit IRC | 15:08 | |
| fungi | Daisy: should be able to go back to using it for now. we're going to figure out the issue by testing between our translate and openstackid dev instances before trying to upgrade openstackid production again | 15:09 |
| Daisy | we have Europe translators. They are in daytime and they are waiting for the server back too. | 15:09 |
| *** IlyaG has joined #openstack-infra | 15:09 | |
| Daisy | Got it, fungi. Thank you. I will let our translators know. | 15:09 |
| *** gongysh has joined #openstack-infra | 15:09 | |
| anteaya | how does patchbot keep getting into #openstack-meeting? | 15:11 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: update nova skipif because of tox name changes https://review.openstack.org/294089 | 15:11 |
| *** dprince has joined #openstack-infra | 15:11 | |
| sdague | anteaya: it's notmyname's right? | 15:11 |
| fungi | smarcet: in the apache error log, it looks like we were getting "Call to a member function postConsent() on a non-object in /srv/openstackid/slot1/app/controllers/openid/UserController.php on line 288" up until a little after 21:00 utc but no sign of that since | 15:11 |
| anteaya | sdague: that is my understanding | 15:11 |
| notmyname | anteaya: I probably forgot to part it yesterday | 15:11 |
| anteaya | and jeblair removed patchbot last time | 15:11 |
| sdague | AJaeger: I think one renaming issue happened - https://review.openstack.org/294089 | 15:11 |
| anteaya | notmyname: jeblair did suggest puppeting it | 15:12 |
| anteaya | notmyname: have you considered that? | 15:12 |
| notmyname | anteaya: I'm not sure what you mean by that | 15:12 |
| anteaya | infra works on the basis that all our services are automated | 15:12 |
| fungi | smarcet: though that looks infrequent so probably just crawlers hitting something normal users don't | 15:13 |
| anteaya | we have puppet modules for all our services, so that we don't have to manually add and remove things | 15:13 |
| AJaeger | sdague: let'S approve https://review.openstack.org/#/c/293637/3/zuul/layout.yaml ;) | 15:13 |
| anteaya | notmyname: jeblair suggested that if you want to add things to infra services you adhere to infra workflow for doing so | 15:13 |
| Zara | (storyboard email sender config still waiting in review: https://review.openstack.org/#/c/293367/ , mentioning again since I think I picked a bad time earlier because of different timezones) | 15:13 |
| *** IlyaG has quit IRC | 15:14 | |
| anteaya | notmyname: we use puppet for the modules and ansible for the orchestration layer | 15:14 |
| notmyname | anteaya: I'm not going to expend any effort to move/run patchbot to -infra. the code's out there if someone else wants too. it's a useful tool I built for me that some other people also find useful | 15:14 |
| anteaya | notmyname: as puppet is not great at orchestration | 15:14 |
| *** Qiming_ has quit IRC | 15:14 | |
| notmyname | anteaya: that being said, I don't see anyone in -meeting complaining about patchbot (ie in the current meeting). is it causing a problem for you in there? | 15:14 |
| anteaya | well we do administer devlopement tools | 15:15 |
| anteaya | including channel administration | 15:15 |
| sdague | AJaeger: sure | 15:15 |
| smarcet | fungi: i think so | 15:15 |
| anteaya | and meeting channels are shared space | 15:16 |
| sdague | AJaeger: https://review.openstack.org/#/c/293526/3 will need skipif changes as well, right? | 15:16 |
| AJaeger | sdague: the change is (wrongly) in https://review.openstack.org/#/c/292472/8/zuul/layout.yaml | 15:16 |
| *** gongysh has quit IRC | 15:16 | |
| AJaeger | So, I'll rebase now and fix... | 15:17 |
| notmyname | anteaya: I'm not asking about the Way Things Are or whatever. is it causing *you* any particular pain? | 15:17 |
| sdague | AJaeger: ok, cool | 15:17 |
| notmyname | anteaya: I've talked to jeblair about it, and I don't have it lurk in any common meeting channel any more. | 15:17 |
| *** dprince has quit IRC | 15:17 | |
| anteaya | I see it as part of my role to proactively notice things that infra may be called on to address | 15:17 |
| *** claudiub|2 has quit IRC | 15:17 | |
| anteaya | and behaviour of meeting channels is one of them | 15:18 |
| nibalizer | fungi: ohai | 15:18 |
| nibalizer | i am running it now | 15:18 |
| nibalizer | im only just now getting back to real wifi | 15:18 |
| fungi | nibalizer: it may take longer than the estimate in the comments when gerrit is under load since it depends a lot on rest api call response times, but shouldn't take more than an hour tops | 15:19 |
| *** jamesmcarthur has quit IRC | 15:19 | |
| nibalizer | i am running it from france too | 15:19 |
| *** jamesmcarthur has joined #openstack-infra | 15:19 | |
| nibalizer | so thats an effect | 15:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 15:20 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs https://review.openstack.org/293526 | 15:20 |
| AJaeger | sdague: rebased, could you review, please ? ^ | 15:20 |
| fungi | nibalizer: hopefully not a huge impact since i did everything i could to batch/reduce individual api calls | 15:20 |
| fungi | but rtt will still play some into it | 15:20 |
| AJaeger | fungi, yolanda it's time now for the final restart of the proposal slave. Could either of you do it, please? | 15:20 |
| yolanda | yep | 15:21 |
| fungi | AJaeger: done | 15:21 |
| fungi | i've been watching/kicking it as i notice it go back offline each time | 15:21 |
| *** moravec has joined #openstack-infra | 15:21 | |
| *** flepied has joined #openstack-infra | 15:21 | |
| yolanda | so what was going on? changes not applying on jjb? | 15:21 |
| fungi | yolanda: the parameter function tells zuul what parameters to include in the gearman work request | 15:22 |
| *** dprince has joined #openstack-infra | 15:22 | |
| AJaeger | yolanda: and it's applied the first time it enters the queue | 15:22 |
| *** pvaneck has joined #openstack-infra | 15:22 | |
| nibalizer | fungi: ok ill keep you posted | 15:23 |
| AJaeger | yolanda: so, the new regex only got applied on *new* jobs but not the once already in periodic queue | 15:23 |
| yolanda | ah | 15:23 |
| fungi | yolanda: so anything _new_ it enqueued in gearman after that config change took effect would no longer get the offline parameter, but jobs it had already queued in gearman still did | 15:23 |
| yolanda | ok understood, and seems there was quite a long queue | 15:23 |
| anteaya | so as a data point we have a whole lot of available nodes, we have 175 available and 350 used: http://grafana.openstack.org/dashboard/db/nodepool | 15:23 |
| fungi | options were to wait it out and keep kicking the worker back online until those flushed through, or restart zuul to clear the gearman queue (which unfortunately would have also lost everything from the post pipeline as well) | 15:24 |
| *** bdemers has joined #openstack-infra | 15:24 | |
| anteaya | and it appears our surplus of available nodes has been evident for about 12 hours | 15:24 |
| *** mrmartin has quit IRC | 15:24 | |
| fungi | anteaya: yes, those are devstack-trusty nodes which aren't getting used but which nodepoold is building due to perceived demand because it still sees gearman registrations for old jobs running on devstack-trusty | 15:24 |
| fungi | anteaya: that will continue until we restart zuul | 15:25 |
| anteaya | ah sorry if this is an old itme | 15:25 |
| anteaya | item | 15:25 |
| anteaya | very good | 15:25 |
| fungi | which will clear out any stale job registrations in gearman | 15:25 |
| AJaeger | we should merge https://review.openstack.org/#/c/293719/ some time today if possible - it moves the old puppet jobs to ubuntu-trusty | 15:25 |
| openstackgerrit | Merged openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes https://review.openstack.org/293637 | 15:25 |
| clarkb | https://review.openstack.org/#/c/293854/2/jenkins/jobs/diskimage-builder.yaml has a slight bug in it in that activate must be sourced | 15:27 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 15:28 |
| *** Sukhdev has quit IRC | 15:28 | |
| AJaeger | clarkb: good find. Will you fix or shall I? | 15:28 |
| clarkb | AJaeger: go for it I am still migrating to proper computer | 15:29 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 15:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix dib virtualenv setup https://review.openstack.org/294097 | 15:30 |
| AJaeger | clarkb: done ^ | 15:30 |
| fungi | smarcet: in the openstackid application log, this debug line looks like all that was being logged when someone would try to authenticate with translate.o.o: http://paste.openstack.org/show/490928/ | 15:30 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 15:30 |
| openstackgerrit | Merged openstack-infra/system-config: Switch translate-dev to openstackid-dev https://review.openstack.org/294060 | 15:30 |
| *** jamesmcarthur has quit IRC | 15:33 | |
| *** angdraug has joined #openstack-infra | 15:33 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 15:36 |
| *** jpr has joined #openstack-infra | 15:36 | |
| *** jamesmcarthur has joined #openstack-infra | 15:36 | |
| *** Hunner has joined #openstack-infra | 15:36 | |
| *** Hunner has quit IRC | 15:36 | |
| *** Hunner has joined #openstack-infra | 15:36 | |
| *** yamahata has quit IRC | 15:37 | |
| *** nwkarsten has joined #openstack-infra | 15:37 | |
| AJaeger | sdague: are the skip-ifs now fine on https://review.openstack.org/#/c/292472/ and https://review.openstack.org/#/c/293526 ? | 15:40 |
| fungi | smarcet: jamesmcarthur: also, the rollback does seem to have gotten authentication for refstack.openstack.org working again as well | 15:40 |
| jamesmcarthur | fungi: indeed - unfortunately we don’t know why | 15:41 |
| *** nwkarste_ has quit IRC | 15:41 | |
| jamesmcarthur | as soon as refstack and translate point to our openstackID-dev server we can sort it out | 15:41 |
| nibalizer | fungi: i think its made it to the 'd' section | 15:41 |
| jamesmcarthur | meanwhile, our apps are broken :\ | 15:41 |
| AJaeger | jamesmcarthur: the change for translate-dev has merged, not sure whether it is deployed already | 15:41 |
| *** Swami has joined #openstack-infra | 15:42 | |
| fungi | AJaeger: it'll still be a few minutes | 15:42 |
| jamesmcarthur | Ajaeger: thanks sir | 15:42 |
| zigo | AJaeger: If I don't use "name", then I use what? | 15:42 |
| jamesmcarthur | we’ll take a look as soon as it’s ready | 15:42 |
| fungi | jamesmcarthur: new apps, or non-backward-compatible lockstep changes between the apps and the openid provider? | 15:42 |
| *** amrith is now known as _amrith_ | 15:43 | |
| zigo | AJaeger: Just: "- pkgdeb-build-pkg" ? | 15:43 |
| jamesmcarthur | new apps | 15:43 |
| fungi | got it | 15:43 |
| AJaeger | zigo: yes | 15:43 |
| zigo | Cheers, doing it. | 15:43 |
| zigo | AJaeger: Same for gate:, I guess, right? | 15:43 |
| AJaeger | zigo: yes | 15:43 |
| zigo | Ok, doing it, thanks. | 15:44 |
| smarcet | fungi: after that we resolve the issue with translate can we redeploy? | 15:44 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 15:44 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Correct branch limitations for ironic-inspector https://review.openstack.org/294106 | 15:44 |
| zigo | fungi: Is it you who uploaded 176625.qcow2? | 15:45 |
| fungi | smarcet: yes, i'm less concerned about refstack at the moment since the real urgency is translators getting their work merged in our very short string freeze window prior to release. if the upgrade re-breaks refstack i think we take our time troubleshooting it by pointing it at openstackid-dev and iterating there (since they don't have a dev deployment they can use) | 15:45 |
| fungi | zigo: i did not, but i asked pabelanger to do so since he was the one keeping an eye on the image builds yesterday | 15:46 |
| zigo | fungi: Ah, thanks so much. I'm nearly done downloading it! :) | 15:46 |
| zigo | It's BIG... | 15:46 |
| *** jpr has quit IRC | 15:47 | |
| fungi | we gave you a heads up on the url to it in channel, but you probably didn't see (it's been busy in here today) | 15:47 |
| fungi | zigo: as soon as diskimage-builder tags a new release, we expect the qcow2 sizes to drop dramatically | 15:47 |
| smarcet | fungi: roger | 15:47 |
| smarcet | :) | 15:47 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 15:47 |
| clarkb | fungi: not really | 15:47 |
| fungi | clarkb: no? | 15:48 |
| clarkb | fungi: the ~5.6GB size is about what we expect. We shouldn't get 11 or 12GB qcow2s anymore though | 15:48 |
| fungi | oh, i thought we were still waiting for the fstrim change in a new release | 15:48 |
| clarkb | fungi: we are, the problem fstrim solves is the 11ish GB image sizes which happen for reasons I still haven't been able to fiure out | 15:49 |
| fungi | also i didn't look at the size of that image, but if it's in the 5-6gb range then that's correct for our images yes | 15:49 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add non voting job to test k8s with dib atomic image https://review.openstack.org/294112 | 15:49 |
| *** sridhar_ram has joined #openstack-infra | 15:49 | |
| clarkb | most of our qcow2s end up in the 6GB range and I don't think we can get them much smaller than that without deleting content | 15:49 |
| *** apoorvad has joined #openstack-infra | 15:49 | |
| fungi | right, that i agree with | 15:49 |
| zigo | It still takes a long time to download 6GB with wget... :P | 15:50 |
| fungi | and i just checked, this image is indeed 5.6G so shouldn't expect it to get any smaller | 15:50 |
| zigo | But I got the image now. | 15:50 |
| *** ybathia has joined #openstack-infra | 15:50 | |
| *** bpokorny has joined #openstack-infra | 15:50 | |
| fungi | yep, i agree it's a big file ;) | 15:50 |
| clarkb | most of that is the git repo cache, then the OS install then the jenkins/devstack images | 15:50 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 15:51 |
| *** piet has quit IRC | 15:51 | |
| *** piet has joined #openstack-infra | 15:51 | |
| clarkb | one of the larger repos is the activity site repo | 15:52 |
| *** nmagnezi has quit IRC | 15:52 | |
| clarkb | but its still less than 200MB | 15:52 |
| clarkb | anteaya: the large number of ready nodes are devstack-trusty hwich are mostly unused at this point (puppet is still using them) | 15:52 |
| *** nadya has quit IRC | 15:53 | |
| zigo | Trying to use kpartx, it doesn't seem to show any partition... | 15:53 |
| anteaya | clarkb: thanks | 15:53 |
| jmccrory | hi all, I have an approved change that depends-on a change in another repo. The depended on change has merged, but I can't seem to get the first to go. Any ideas on how to get it moving? | 15:53 |
| clarkb | zigo: there should be a single / ext4 filesystem | 15:53 |
| anteaya | puppet patches should be happy right now | 15:53 |
| clarkb | zigo: not sure how that ends up in a partition table with qcow2 | 15:53 |
| zigo | clarkb: So I can just mount it using loopback? | 15:53 |
| anteaya | jmccrory: link to patch in question? | 15:53 |
| AJaeger | jmccrory: somebody needs to approve it again - or toggle the +A. | 15:53 |
| jmccrory | anteaya: https://review.openstack.org/#/c/289510/ | 15:53 |
| clarkb | zigo: yes, do whatever the dance is for qcow2 images | 15:54 |
| AJaeger | jmccrory: http://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats | 15:54 |
| *** jpr has joined #openstack-infra | 15:54 | |
| zigo | clarkb: oh, I'm silly, I should convert it first to a raw format using qemu-utils ... | 15:54 |
| jmccrory | AJaeger: ah ok, thanks! | 15:54 |
| anteaya | jmccrory: as AJaeger said | 15:54 |
| anteaya | jmccrory: the dependency was +A'd before the depends-on was merged | 15:55 |
| clarkb | sdague: were you going to write the pkg-map file for the uuid-runtime change? should someone else push that patch? | 15:55 |
| jmccrory | anteaya: yep looks like that's what happened, thanks. missed that section when reading through earlier | 15:55 |
| *** krtaylor has quit IRC | 15:55 | |
| anteaya | jmccrory: a new +A on the dependency is needed for zuul to see that as a trigger, since at the time of the first +A the conditions weren't met | 15:55 |
| anteaya | jmccrory: yup, zuul has no concept of time, it lives in the ever present now | 15:56 |
| sdague | clarkb: it was not on my radar, as I'm in another stack of code | 15:56 |
| anteaya | it is a rather zen tool | 15:56 |
| zigo | BAM, 17 GB... :P | 15:56 |
| openstackgerrit | Merged openstack-infra/project-config: Fix dib virtualenv setup https://review.openstack.org/294097 | 15:58 |
| zigo | Fun, you guys have your ssh keys in a $home! :) | 15:59 |
| *** bmjen has joined #openstack-infra | 15:59 | |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config: oslo.db: don't run tests with SQLAlchemy 0.9.x https://review.openstack.org/294122 | 16:00 |
| fungi | zigo: yeah, that's one of the reasons those images aren't generally reusable | 16:00 |
| *** placko_ has quit IRC | 16:00 | |
| *** jamesmcarthur has quit IRC | 16:01 | |
| zigo | fungi: Jenkins currently complains about a syntax error in jenkins/scripts/propose_translation_update.sh, which I didn't even touch... | 16:01 |
| *** IlyaG has joined #openstack-infra | 16:02 | |
| zigo | [E] E040: Syntax error: syntax error near unexpected token `fi': '' | 16:02 |
| fungi | AJaeger: ^ known issue with the translation update script? | 16:02 |
| clarkb | greghaynes: for dibs pkg-map stuff is there are precanned map? I cannot find one anywhere so guessing not? | 16:02 |
| AJaeger | Let me check... | 16:02 |
| AJaeger | zigo, have a link for me? | 16:02 |
| *** adrian_otto has joined #openstack-infra | 16:03 | |
| zigo | AJaeger: http://logs.openstack.org/22/294022/7/check/gate-project-config-linters/95f7207/console.html | 16:03 |
| abregman | hi everyone. I asked this before, but I'll try again: any tips for openstack-infra newbie on how to start? (beside reviewing) | 16:03 |
| AJaeger | zigo: you misparsed bashate ;( | 16:03 |
| AJaeger | "- /home/jenkins/workspace/gate-project-config-linters/jenkins/scripts/pkgdeb-install-sbuild.sh : L88" | 16:03 |
| AJaeger | that's the error | 16:03 |
| zigo | Oh... | 16:03 |
| zigo | Thanks. | 16:03 |
| *** mattymo_ is now known as mattymo | 16:04 | |
| *** samuelBartel has quit IRC | 16:05 | |
| anteaya | abregman: hello, well infra is large and takes a long time to learn, so welcome | 16:05 |
| anteaya | abregman: and as you are learning patience is indispensible here | 16:05 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 16:05 |
| *** arxcruz has quit IRC | 16:06 | |
| anteaya | abregman: have you any interest in any aspect of infra that you have seen? | 16:06 |
| abregman | anteaya: thank you. any suggestions on where to start? | 16:06 |
| anteaya | abregman: it is often helpful to start with one tool and then work from there | 16:06 |
| *** zeih has quit IRC | 16:06 | |
| abregman | anteaya: would tool considered as a good start for beginners? | 16:06 |
| anteaya | abregman: do you have any prior experience with any deployment tools, sysadmin, code review tools | 16:06 |
| anteaya | abregman: that depends on you | 16:07 |
| *** tphummel has quit IRC | 16:07 | |
| anteaya | abregman: you at least can irc I see | 16:07 |
| anteaya | what is easy for one person is hard for another | 16:07 |
| anteaya | is there any aspect that you see that you like yet? | 16:07 |
| anteaya | I'm trying to help, but it isn't a one size fits all answer | 16:08 |
| pabelanger | and back | 16:08 |
| abregman | anteaya: I have experience as sys admin and python developer. I'm familiar with Ansible, Jenkins, Python, Linux :) | 16:08 |
| anteaya | awesome | 16:08 |
| *** jamesmcarthur has joined #openstack-infra | 16:08 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch-dsvm job https://review.openstack.org/294128 | 16:08 |
| smarcet | fungi:Ajaeger: the proposed change on https://review.openstack.org/#/c/294060/1/manifests/site.pp is wrong | 16:08 |
| anteaya | what experience have you with jenkins things? | 16:08 |
| smarcet | fungi: the rite uri is https://openstackid-dev.openstack.org/ | 16:08 |
| anteaya | are you familiar at all with gearman? | 16:08 |
| anteaya | pabelanger: welcome back | 16:08 |
| AJaeger | smarcet: OOpos ;( | 16:08 |
| AJaeger | smarcet: will fix... | 16:08 |
| smarcet | ok cool thx :) | 16:08 |
| fungi | oh, yep--sorry for not spotting that | 16:09 |
| *** aeng has joined #openstack-infra | 16:09 | |
| smarcet | np :) | 16:09 |
| *** mylu has joined #openstack-infra | 16:09 | |
| *** mattt has left #openstack-infra | 16:09 | |
| *** krtaylor has joined #openstack-infra | 16:09 | |
| aeng | ping Daisy | 16:09 |
| abregman | anteaya: I setup several jobs that deploys OpenStack using packstack (by writing jjbs) in various topologies | 16:09 |
| Daisy | wow | 16:09 |
| anteaya | abregman: awesome | 16:09 |
| Daisy | aeng: Hi | 16:09 |
| pabelanger | AJaeger: do you mind helping 293767 land? I'd like to do some testing on debian-jessie with puppet apply jobs | 16:09 |
| aeng | :), just saw your message | 16:09 |
| abregman | anteaya: I'm not familiar with gearman | 16:10 |
| anteaya | abregman: okay good | 16:10 |
| Daisy | AJaeger, fungi : aeng is Alex Eng from Zanata dev team. | 16:10 |
| *** vgridnev has quit IRC | 16:10 | |
| pabelanger | fungi: or if you have a sec for 293767 | 16:10 |
| aeng | looks like the login works now | 16:10 |
| *** zeih has joined #openstack-infra | 16:10 | |
| Daisy | aeng: It's your midnight, I think. | 16:10 |
| aeng | yup, 2am | 16:10 |
| anteaya | abregman: have you seen this image before? http://docs.openstack.org/infra/manual/developers.html#quick-reference | 16:10 |
| Daisy | aeng: yes. fungi and his team rollback the openstack id. | 16:11 |
| aeng | oh... so in fact some changes happened in openid | 16:11 |
| abregman | anteaya: yes and I wrote this: http://abregman.com/2016/03/05/openstack-infra-jenkins-jobs/ | 16:11 |
| *** mylu has quit IRC | 16:11 | |
| Daisy | aeng: I think they want your help to make the new OpenStack ID version workable. | 16:11 |
| aeng | ah... | 16:11 |
| abregman | anteaya: I simply don't know what to work on :) there are so many projects | 16:12 |
| aeng | Daisy, even if we made the changes, it wont deploy soon.. we need to coordinate the deployment date | 16:12 |
| fungi | Daisy: aeng: yeah, we're working on getting our translate-dev site authenticating against the dev deployment of our openid provider so we can troubleshoot further while not blocking the translation team | 16:12 |
| anteaya | abregman: yup, I am trying to figure out how to help you | 16:13 |
| fungi | aeng: i think we're mostly just looking for help/insight into what the openid consumer implementation in zanata expects so we can make sure we cater to that in our provider implementation | 16:13 |
| anteaya | abregman: have you seen this before? http://status.openstack.org/zuul/ | 16:13 |
| aeng | fungi, good point | 16:13 |
| * aeng checking now | 16:13 | |
| yolanda | AJaeger, just to confirm, any job that ends with -nv is non-voting right? no need to specify it | 16:13 |
| pabelanger | anteaya: AJaeger okay, approved 293719 and will be monitoring JJB | 16:13 |
| abregman | anteaya: yes I did | 16:14 |
| AJaeger | yolanda: corrrect | 16:14 |
| fungi | aeng: since some thought to be non-impacting adjustments between openstackid 1.0.13 and 1.0.14 seem to have caused zanata to no longer be able to authenticate against it | 16:14 |
| yolanda | ++ | 16:14 |
| greghaynes | clarkb: There arent really pre-canned maps, no | 16:14 |
| clarkb | greghaynes: and behavior if one distro doesn't have the package name is to skip or fail? | 16:14 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Fix openstack-id location for translation-dev https://review.openstack.org/294131 | 16:15 |
| greghaynes | clarkb: hah, the behavior is itll try and install the package. Whether that succeeds or fails is up to the package installer | 16:15 |
| *** zeih has quit IRC | 16:15 | |
| anteaya | pabelanger: ack | 16:15 |
| clarkb | fun | 16:15 |
| *** MarkAtwood has joined #openstack-infra | 16:15 | |
| clarkb | so ya we should write a pkg-map to avoid surprises | 16:15 |
| anteaya | pabelanger: ping if you spot a bug and need help with a fix | 16:15 |
| AJaeger | smarcet, fungi : 294131 is the fix - is it fine now? | 16:15 |
| greghaynes | clarkb: yep | 16:15 |
| anteaya | abregman: yes, I got to that place in your blog post, nicely written | 16:15 |
| pabelanger | anteaya: will do | 16:15 |
| abregman | anteaya: thank you | 16:16 |
| fungi | AJaeger: yes, that's the correct domain name | 16:16 |
| anteaya | abregman: okay, so as you surmised one of the hardest things is to have an overview | 16:16 |
| fungi | AJaeger: should be able to load it in a browser to confirm too | 16:16 |
| AJaeger | fungi: confirmed, I can load it | 16:16 |
| AJaeger | pabelanger: thanks for taking care of the puppet devstack-trusty | 16:17 |
| *** vgridnev has joined #openstack-infra | 16:17 | |
| anteaya | abregman: while it seems like it might be a whole lot of work for not much gain, personally what I would give you as a first assignment is read the scrollback for the past two days and give me a summary of what you think were items of importance that came up and were acted on | 16:17 |
| *** vgridnev has quit IRC | 16:17 | |
| aeng | fungi, all I can tell is that we are using generic openid impl | 16:17 |
| smarcet | AJaeger: yes its is | 16:18 |
| smarcet | thx | 16:18 |
| anteaya | abregman: we have a lot of people working on a lot of different things, but the amount of people who are understanding how the pieces all fit together and co-ordinate are small | 16:18 |
| AJaeger | pabelanger: 293767 will conflict with the puppet change you just approved. Once it's rebased, I'll help land it... | 16:18 |
| pabelanger | AJaeger: ya, I am fine with that | 16:18 |
| AJaeger | but now I need to step out for two hours... | 16:18 |
| anteaya | abregman: a large part of our work is seeing what needs doing and doing it, we don't have time to give folks direction | 16:18 |
| aeng | fungi, what changes in the upgrades? | 16:19 |
| *** nwkarsten has quit IRC | 16:19 | |
| abregman | anteaya: just to understand I get it right, you mean I should read the history of this irc channel for the past two days? | 16:19 |
| anteaya | abregman: now, learning how to parse the backscroll and interpret it can take time, but I think that would be time well spent and I'm willing to hellp you with that | 16:19 |
| anteaya | abregman: yes, that is what I am saying | 16:19 |
| *** ybathia has quit IRC | 16:19 | |
| anteaya | abregman: now if you don't want to do that, the other option is stand up a gerrit on your own | 16:19 |
| anteaya | abregman: to have that experience | 16:20 |
| abregman | anteaya: k then! let's do it! | 16:20 |
| anteaya | abregman: awesome | 16:20 |
| anteaya | abregman: do you know how to find the logs? | 16:20 |
| abregman | anteaya: first question - where can I read all the channel history? | 16:20 |
| abregman | nope | 16:20 |
| *** nwkarsten has joined #openstack-infra | 16:20 | |
| anteaya | abregman: great question | 16:20 |
| anteaya | abregman: http://eavesdrop.openstack.org/ | 16:20 |
| *** moravec has quit IRC | 16:20 | |
| anteaya | that is the link to all the logs for all of our channels as well as all our meeting times | 16:21 |
| *** jamesmcarthur has quit IRC | 16:21 | |
| fungi | aeng: short changelog is readable at https://git.openstack.org/cgit/openstack-infra/openstackid/tag/?h=1.0.14 | 16:21 |
| anteaya | abregman: see the link " IRC Channel Logs"? | 16:21 |
| *** hasharMeeting has quit IRC | 16:21 | |
| anteaya | abregman: you should see an index with #openstack-infra listed | 16:21 |
| abregman | anteaya: yes | 16:21 |
| anteaya | abregman: link me when you find the infra logs | 16:21 |
| abregman | anteaya: found it | 16:22 |
| fungi | aeng: the 6 commits which implemented that can be seen linked from the top of the list at https://git.openstack.org/cgit/openstack-infra/openstackid/log/?h=1.0.14 | 16:22 |
| anteaya | abregman: great | 16:22 |
| *** pcaruana has joined #openstack-infra | 16:22 | |
| anteaya | abregman: this is where you get to work on your patience and persistence skills | 16:22 |
| *** doug-fish has quit IRC | 16:22 | |
| jpr | abergman, anteaya: I've been dormant for a many weeks (other work...feh) but this is where I left off. standing up a gerrit. would feel good to jump in an collaborate on overview and sharing other notes from my work in jan. | 16:22 |
| openstackgerrit | Merged openstack-infra/project-config: Move puppet jobs to ubuntu-trusty https://review.openstack.org/293719 | 16:22 |
| anteaya | abregman: ask questions as you have them, use paste and etherpad and the channel log urls to ensure I understand exactly what item you are referenceing when you reference something | 16:22 |
| anteaya | abregman: and again, welcome | 16:23 |
| *** doug-fish has joined #openstack-infra | 16:23 | |
| anteaya | jpr: awesome, welcome | 16:23 |
| *** Daisy has quit IRC | 16:23 | |
| abregman | anteaya: great! what do you usually use to paste text and share it? | 16:23 |
| anteaya | jpr: I didnt' set abregman's first task as gerrit mostly due to my percieved lack of people available to respond to questions as they arise | 16:24 |
| *** nwkarste_ has joined #openstack-infra | 16:24 | |
| openstackgerrit | Dmitry Tyzhnenko proposed openstack-infra/git-review: Add reviewers by group alias on upload https://review.openstack.org/195043 | 16:24 |
| anteaya | abregman: paste.openstack.org | 16:24 |
| *** andymaier has joined #openstack-infra | 16:24 | |
| jpr | anteaya: yes, gerrit is not the place to start. :) That's where I left off. :) | 16:24 |
| anteaya | jpr abregman but if there are two of you, that is an ideal situation since I find a great first learning step is to compare notes with others | 16:24 |
| *** ifarkas_ has quit IRC | 16:25 | |
| *** nwkarsten has quit IRC | 16:25 | |
| anteaya | jpr: do you want to work on the same task I set for abregman? | 16:25 |
| jpr | anteaya: agreed | 16:25 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 16:25 |
| anteaya | jpr abregman and then at the very least, you have each other to talk to? | 16:25 |
| anteaya | and the rest of us can support as we have time? | 16:25 |
| AJaeger | could some project-config cores please review https://review.openstack.org/292472 - to start with the python-db migration, please? | 16:25 |
| abregman | yup. hi jpr :) what is your timezone? | 16:25 |
| *** salv-orlando has quit IRC | 16:25 | |
| anteaya | AJaeger: looking | 16:26 |
| jpr | anteaya: ragarding task, i'd like to get my notes squared up and shared on getting started. I ran into a lot of little things that took some debugging to understand | 16:26 |
| *** salv-orlando has joined #openstack-infra | 16:26 | |
| * AJaeger steps out... | 16:26 | |
| AJaeger | thanks, anteaya | 16:26 |
| jpr | abergman: cst (central us) | 16:26 |
| *** tphummel has joined #openstack-infra | 16:26 | |
| abregman | anteaya: thank you for my first assignment! /me exicted. Once I finish it, I'll ping you | 16:27 |
| anteaya | AJaeger: awesome so mriedem is already aware of this | 16:27 |
| anteaya | abregman: oh don't wait until you finish, ask as you go along | 16:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: add uuid-runtime to base image builds https://review.openstack.org/293942 | 16:27 |
| clarkb | greghaynes: ^ did I do that right? | 16:27 |
| *** doug-fish has quit IRC | 16:27 | |
| anteaya | jpr: as you are comfortable, everyone learns with a different style | 16:27 |
| abregman | jpr: mine is ist (Israel) | 16:27 |
| *** fernnest has quit IRC | 16:28 | |
| *** jamesmcarthur has joined #openstack-infra | 16:28 | |
| anteaya | jpr abregman I'm just thinking how fortuitous it is that you both are here, as I find learning with someone at the same level really helps both learning and retention | 16:28 |
| anteaya | so thank you both | 16:28 |
| aeng | fungi, I will need to check with the team tomorrow for more details and let you guys know what is expected from Zanata side. | 16:28 |
| *** ddieterly is now known as ddieterly[away] | 16:29 | |
| aeng | whats the best way to communicate this with you? | 16:29 |
| abregman | thank you anteaya! | 16:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add experimental puppet-apply job for debian-jessie https://review.openstack.org/293767 | 16:29 |
| fungi | aeng: thanks. we're going to continue troubleshooting from the provider end as well and try to spot the mismatch (or worst case bisect changes until we pull out the offending one) | 16:29 |
| abregman | jpr: what do you plan to do now? | 16:29 |
| pabelanger | AJaeger: rebased ^ | 16:29 |
| jpr | anteaya: yes. an fortuitous that I was itching to read the IRC channel again at this time. :) | 16:29 |
| anteaya | AJaeger: nit, you have pep8, docs in check and docs, pep8 in gate, I know you would care about having consistent order in the python-db-jobs zuul template | 16:30 |
| fungi | aeng: you can get up with us in here or on the openstack-infra@lists.openstack.org mailing list | 16:30 |
| *** mikelk has quit IRC | 16:30 | |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack: Make puppet pull releases from Pypi https://review.openstack.org/281737 | 16:30 |
| aeng | cool | 16:30 |
| anteaya | abregman jpr wonderful! :) | 16:30 |
| *** rhallisey has quit IRC | 16:30 | |
| jpr | abergman: i'm going to review your blog and then start pulling together my notes and experience from a few weeks ago. I have a working (then) pipeline of zuul, nodepool, jenkins | 16:30 |
| *** rhallisey has joined #openstack-infra | 16:31 | |
| *** nwkarste_ has quit IRC | 16:31 | |
| *** ddieterly[away] is now known as ddieterly | 16:31 | |
| abregman | jpr: cool. I'm going to work on Anita task. probably will have a ton of questions on the way. will try to summarize it in the blog for newbies as me. | 16:32 |
| *** doug-fish has joined #openstack-infra | 16:38 | |
| anteaya | AJaeger: +2 on 292472, if you decide to address the nit, will +2 again | 16:38 |
| *** doug-fish has quit IRC | 16:38 | |
| *** doug-fish has joined #openstack-infra | 16:38 | |
| *** lakshmiS has joined #openstack-infra | 16:39 | |
| jpr | anita | 16:39 |
| jpr | oops ignore | 16:39 |
| pabelanger | clarkb: fungi what is the preferred way to delete all the ubuntu-trusty ready nodes in nodepool these days? we are at min-ready 1 but have 130 currently ready | 16:39 |
| pabelanger | clarkb: fungi: I assume nodepool delete | 16:40 |
| clarkb | pabelanger: you mean devstack-trusty? they will accumulate until we switch puppet away from them and restart zuul/geard | 16:40 |
| pabelanger | clarkb: sorry, have devstack-trusty | 16:40 |
| harlowja_at_home | clarkb https://review.openstack.org/#/c/294147/ as requested | 16:40 |
| fungi | pabelanger: we need a zuul restart or they'll keep coming back | 16:40 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Make ubuntu-core support releases https://review.openstack.org/294149 | 16:40 |
| clarkb | pabelanger: first step is get puppet switched, but wanted to make sure we were relatively stable with other things before we do that, then switch puppet and restart zuul | 16:41 |
| pabelanger | clarkb: fungi: Ya, JJB running right now for puppet switch | 16:41 |
| pabelanger | currently monitoring | 16:41 |
| lakshmiS | we at searchlight project need elasticsearch installed for our functional tests to run on zuul. How do we get elasticsearch running on zuul test environment? | 16:43 |
| clarkb | lakshmiS: install it as part of your test jobs | 16:43 |
| lakshmiS | Is there an example on any other software like mysql being installed like that? | 16:44 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 16:45 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 16:45 | |
| *** scheuran has quit IRC | 16:45 | |
| *** doug-fish has quit IRC | 16:45 | |
| clarkb | lakshmiS: yes, postgres and mysql are both installed as part of the jobs now. In devstack it installs all of those deps, for unittests the jenkins job config does it | 16:46 |
| *** doug-fish has joined #openstack-infra | 16:46 | |
| *** doug-fish has quit IRC | 16:46 | |
| clarkb | lakshmiS: do your functional tests rely on devstack? if so I would have it install elasticsearch for you via their plugin mechanism | 16:46 |
| *** doug-fish has joined #openstack-infra | 16:46 | |
| lakshmiS | no it doesn't rely on devstack | 16:47 |
| fungi | argh! why are we getting NOT_REGISTERED on gate-openstackci-beaker-devstack-centos7, gate-openstackci-beaker-ubuntu-trusty and gate-infra-puppet-apply-devstack-centos7 now? | 16:48 |
| clarkb | fungi: zuul updated before jenkins is my guess | 16:48 |
| fungi | i'm diving into zuul logs/gearman queue | 16:49 |
| fungi | yeah, but updated what? did we switch nodes for those? | 16:49 |
| clarkb | fungi: apparently my change merged this morning yes | 16:49 |
| fungi | like in the past few minutes? | 16:49 |
| *** mriedem1 has joined #openstack-infra | 16:49 | |
| clarkb | ya 16:41:38 pabelanger | clarkb: fungi: Ya, JJB running right now for puppet switch | 16:49 |
| *** jlanoux has quit IRC | 16:49 | |
| *** mriedem has quit IRC | 16:49 | |
| fungi | there it is. 293719 merged at 1622 | 16:50 |
| lakshmiS | clarkb: can you suggest a project where func tests are using devstack. maybe i can model it based on that? | 16:50 |
| *** esikachev has quit IRC | 16:50 | |
| clarkb | lakshmiS: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n619 is what the non devstack tests do, they install packages listed by bindep in other-requirements.txt | 16:50 |
| *** electrofelix has quit IRC | 16:51 | |
| fungi | pabelanger: so the names of those jobs changed? | 16:51 |
| clarkb | lakshmiS: unfortunately elasticsearch isn't in a lot of distros last I checked so your install stuff will have to be smarter | 16:51 |
| clarkb | fungi: yes because the old job setup was well bad | 16:51 |
| clarkb | fungi: they relied on "ostype" and assumed that that could be appended to "devstack" | 16:51 |
| pabelanger | what clarkb said | 16:52 |
| fungi | okay. looks like the devstack-centos7 ones haven't registered yet but the ubuntu-trusty one has now | 16:52 |
| clarkb | fungi: which meant the switch away from devstack-trusty required changing all sorts of stuff | 16:52 |
| clarkb | because that was a bad built in assumption that no longer holds | 16:52 |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| fungi | i'll see if i can urge nodepool to built more devstack-centos7 quickly | 16:52 |
| *** smarcet has quit IRC | 16:52 | |
| fungi | ahh, there they go | 16:52 |
| anteaya | yay | 16:52 |
| *** nadya has joined #openstack-infra | 16:52 | |
| fungi | okay, all set to recheck things now | 16:52 |
| *** esikachev has joined #openstack-infra | 16:52 | |
| anteaya | so it was a race? not an issue with the code? | 16:53 |
| clarkb | a couple jobs still use "ostype" for the internal beaker config, we will need to update all the projects to use less implicit node info | 16:53 |
| *** nadya has quit IRC | 16:53 | |
| clarkb | anteaya: yes I think so. If zuul updates before jenkins does then this happens | 16:53 |
| *** ddieterly[away] is now known as ddieterly | 16:53 | |
| anteaya | okay | 16:53 |
| lakshmiS | clarkb: thanks for the pointers. | 16:54 |
| anteaya | does ansible track the jenkins zuul update order at all? | 16:54 |
| clarkb | anteaya: we don't force zuul after jenkins but zuul by virtue of starting with a z seems to go last consistently | 16:54 |
| *** vincentll has quit IRC | 16:55 | |
| *** aeng has quit IRC | 16:55 | |
| anteaya | wonder what happened this time | 16:55 |
| clarkb | anteaya: the larger issue is when doing a lot of job updates puppet times out bfeore all the things update then zuul updates | 16:55 |
| pabelanger | fungi: Ya, jjb still making the rounds. | 16:55 |
| *** placko_ has joined #openstack-infra | 16:55 | |
| anteaya | z still happens after j for my alphabet | 16:55 |
| *** placko_ has quit IRC | 16:55 | |
| anteaya | ah the timeout | 16:55 |
| clarkb | anteaya: it does but due to the timeouts we end up stopping the update on jenkins then running zuuls update | 16:55 |
| anteaya | yup okay | 16:55 |
| anteaya | thanks for the explanation | 16:56 |
| nibalizer | fungi: it finished! | 16:58 |
| pabelanger | 15mins in and JJB still running on jenkins05 (plus others) | 16:59 |
| fungi | nibalizer: note that second run (it will be quicker) to generate the stable maint electorate until i or someone gets a chance to figure out a better way to make that happen as part of the same run | 16:59 |
| *** madhu_ak has joined #openstack-infra | 17:00 | |
| nibalizer | fungi: o7 | 17:00 |
| anteaya | nibalizer: oh you are creating the electoral rolls | 17:01 |
| pabelanger | okay, looks like JJB failed on jenkins | 17:01 |
| anteaya | nibalizer: nice | 17:01 |
| pabelanger | so, what is the best way to manually run it without having puppet collide | 17:01 |
| *** armax has quit IRC | 17:02 | |
| fungi | who can help me interpret this new lovely failure? https://jenkins04.openstack.org/job/gate-openstackci-beaker-ubuntu-trusty/1/console | 17:02 |
| *** ddieterly is now known as ddieterly[away] | 17:02 | |
| clarkb | pabelanger: not sure I understand the question | 17:02 |
| nibalizer | anteaya: yip | 17:03 |
| fungi | "Error: Could not find a suitable provider for cron" | 17:03 |
| clarkb | fungi: wow ubuntu-minimal doesn't have cron | 17:03 |
| nibalizer | nice | 17:03 |
| *** tphummel_ has joined #openstack-infra | 17:03 | |
| clarkb | maybe we should revert everythin and go back to precomputer society | 17:03 |
| anteaya | nibalizer: thank you | 17:03 |
| anteaya | clarkb: I like rocks and sticks | 17:03 |
| bkero | "Error: Could not find a suitable provider for cron" | 17:03 |
| bkero | fungi: ^ | 17:03 |
| fungi | i'm going to go ahead and force merge 294131 for now so we can get on with the zanata/openstackid troubleshooting while people work through the job change failures | 17:04 |
| *** yamamoto has quit IRC | 17:04 | |
| anteaya | fungi: ack | 17:04 |
| pabelanger | clarkb: I want to manually run JJB. Do I need to stop anything on puppetmaster? | 17:04 |
| pabelanger | for jenkins.o.o for example | 17:04 |
| clarkb | pabelanger: no you should be fine to just run it | 17:04 |
| pabelanger | ok | 17:04 |
| *** tphummel has quit IRC | 17:04 | |
| *** tphummel_ is now known as tphummel | 17:04 | |
| fungi | bkero: yep, i think we switched to images that don't have cron installed by default and the manifest doesn't install it | 17:04 |
| *** Sukhdev has joined #openstack-infra | 17:04 | |
| anteaya | clarkb: so what do you think the best solution is to addressing the no cron issue? | 17:04 |
| *** Daisy has joined #openstack-infra | 17:04 | |
| pabelanger | clarkb: sudo jenkins-jobs update .. is fine? | 17:05 |
| bkero | fungi: sounds like something should install cron :) | 17:05 |
| *** mriedem1 is now known as mriedem | 17:05 | |
| clarkb | pabelanger: yup, except sudo likely to have cache problems, I tend to su to not worry about the env | 17:05 |
| pabelanger | ok | 17:05 |
| clarkb | anteaya: the "best" solution imo is to make ubuntu-minimal stop being so minimal | 17:05 |
| fungi | or sudo -H | 17:05 |
| anteaya | clarkb: revert? fixing the image and building new images will take a while | 17:05 |
| clarkb | but everytime I suggest that I get told that ins't minimal | 17:05 |
| clarkb | anteaya: yes that may be the pragmatic solution for now | 17:06 |
| anteaya | clarkb: ah, I didn't realize there was some prior conversation here | 17:06 |
| anteaya | clarkb: do you know which patch needs reverting? | 17:06 |
| clarkb | anteaya: well previously we hav easked for sshd to be installed | 17:06 |
| pabelanger | doing jenkins05 atm | 17:06 |
| *** jamesmcarthur has quit IRC | 17:06 | |
| clarkb | anteaya: https://review.openstack.org/#/c/293719/ is the change | 17:06 |
| anteaya | clarkb: do you want to offer the revert? | 17:07 |
| pabelanger | and jenkins.o.o | 17:07 |
| *** vgridnev has joined #openstack-infra | 17:07 | |
| anteaya | clarkb: wondered if there was anything in addition to your change | 17:07 |
| openstackgerrit | Merged openstack-infra/system-config: Fix openstack-id location for translation-dev https://review.openstack.org/294131 | 17:07 |
| clarkb | anteaya: mostly trying to figure out if there is a sane alternative | 17:07 |
| anteaya | clarkb: yup, standing by | 17:07 |
| clarkb | could add cron to sdagues change and rebuild really quick | 17:08 |
| anteaya | clarkb: if you think that is worth a try | 17:08 |
| anteaya | which change of sdague's? | 17:08 |
| anteaya | 293942? | 17:08 |
| *** Daisy has quit IRC | 17:09 | |
| clarkb | ya thats the one | 17:09 |
| anteaya | k | 17:10 |
| *** madhu_ak has quit IRC | 17:10 | |
| *** Sukhdev has quit IRC | 17:10 | |
| anteaya | sdague: you about? we seem to be missing cron on ubuntu-minimal, clarkb is wondering if we can add install cron on your patch 293942 | 17:10 |
| anteaya | he might be offline for a bit, he said he was working on something earlier | 17:12 |
| anteaya | clarkb: you want to wait? | 17:12 |
| *** gsilvis has joined #openstack-infra | 17:12 | |
| pabelanger | or adding it to bindep-fallback.txt for the moment too | 17:12 |
| pabelanger | but that needs a rebuild too | 17:13 |
| fungi | pabelanger: that won't help. it's not used for those jobs anyway | 17:13 |
| pabelanger | Ah | 17:13 |
| *** jamesmcarthur has joined #openstack-infra | 17:13 | |
| fungi | bindep-fallback.txt is basically providing a default package set for jobs which were previously running on bare-trusty | 17:14 |
| anteaya | right now I'm in the revert camp, if anyone is counting | 17:14 |
| fungi | these jobs were running on devstack-trusty | 17:14 |
| pabelanger | right | 17:14 |
| fungi | and yeah, i think that job update needs to be undone in the short term | 17:14 |
| pabelanger | anteaya: revert which? | 17:14 |
| anteaya | pabelanger fungi thanks I didn't know that | 17:14 |
| anteaya | revert https://review.openstack.org/#/c/293719/ | 17:14 |
| anteaya | as that is the patch that moved the jobs | 17:15 |
| pabelanger | Oh, that's the one I am monitoring ATM | 17:15 |
| anteaya | yeah | 17:15 |
| pabelanger | so, jenkins.o.o and jenkins05 are currently updating | 17:15 |
| anteaya | and as fungi pointed out, we lack cron | 17:15 |
| *** nmagnezi has joined #openstack-infra | 17:15 | |
| pabelanger | odd it took this long to fine it | 17:15 |
| *** esikachev has quit IRC | 17:16 | |
| pabelanger | so, discussion is to add it to ubuntu-minimal in diskimage-builder ATM right? | 17:16 |
| fungi | pabelanger: yeah, apparently puppet-openstackci sets up cron jobs but doesn't ensure cron is installed | 17:16 |
| pabelanger | or we update nodepool element in system-config | 17:16 |
| anteaya | so right now the options are revert or, add cron and rebuild images | 17:16 |
| openstackgerrit | Sam Betts proposed openstack-infra/glean: Enable multiple VLANs per physical interface https://review.openstack.org/283726 | 17:16 |
| openstackgerrit | Sam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces https://review.openstack.org/293648 | 17:16 |
| *** fawadkhaliq has joined #openstack-infra | 17:16 | |
| *** Sukhdev has joined #openstack-infra | 17:16 | |
| fungi | it's possible we could add a cron package resource to that manifest | 17:16 |
| anteaya | fungi: that won't require updating images, correct? | 17:17 |
| pabelanger | Or have puppet team update their manifests to ensure crontab is installed | 17:17 |
| *** erlon has quit IRC | 17:17 | |
| pabelanger | like we did for locale issue | 17:17 |
| clarkb | fungi: I don't know if puppet will work on a first pass if it installs cron | 17:17 |
| fungi | which would also ensure that it works for people who apply the openstackci module on a server which didn't have cron preinstalled | 17:17 |
| *** Sukhdev has quit IRC | 17:17 | |
| clarkb | that will not work for pip iirc | 17:17 |
| pabelanger | fungi: right, I think that is the better solution | 17:17 |
| clarkb | or package manager resources | 17:17 |
| *** nadya has joined #openstack-infra | 17:18 | |
| bkero | You could always if (not defined(Package['cron'])) { package { "cron": ensure => installed } } | 17:18 |
| *** sc68cal has joined #openstack-infra | 17:18 | |
| pabelanger | okay, holding off on other jenkins masters until we decide to revert or not | 17:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially https://review.openstack.org/292472 | 17:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs https://review.openstack.org/293526 | 17:18 |
| clarkb | bkero: right but does that require two puppet runs? I think it does | 17:18 |
| anteaya | pabelanger: thanks | 17:18 |
| pabelanger | bkero: which job is failing? | 17:18 |
| bkero | clarkb: nope | 17:18 |
| anteaya | pabelanger: https://jenkins04.openstack.org/job/gate-openstackci-beaker-ubuntu-trusty/1/console | 17:19 |
| clarkb | bkero: are you sure? I know that we couldn't bootstrap pip that way which is why pip is installed out of band | 17:19 |
| *** sputnik13 has joined #openstack-infra | 17:19 | |
| bkero | clarkb: you put a requirement/ordering in it | 17:19 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 17:19 |
| bkero | So you have cron { "Mycronthing": requires => Package['cron'] } in there too | 17:19 |
| *** fernnest has joined #openstack-infra | 17:19 | |
| fungi | oh, right, the cron module probably needs to be the one to make sure cron is installed | 17:19 |
| AJaeger | anteaya: updated https://review.openstack.org/#/c/292472/ as you suggested, could you review again, please? | 17:19 |
| pabelanger | right, I think that is the correct fix. Especially since we are talking about puppet failing | 17:20 |
| *** esikachev has joined #openstack-infra | 17:20 | |
| pabelanger | we basically exposed a bug in the manifest | 17:20 |
| anteaya | AJaeger: done, thank you | 17:20 |
| AJaeger | could I get some review love on https://review.openstack.org/#/c/285519/ , please? This publishes barbican api guide. I did the last few iterations of it, thus only a +1 from me. | 17:20 |
| *** fawadkhaliq has quit IRC | 17:20 | |
| bkero | pabelanger: some might say we exposed a bug in ubuntu's image :) | 17:21 |
| clarkb | bkero: its our image | 17:21 |
| *** fawadkhaliq has joined #openstack-infra | 17:21 | |
| clarkb | bkero: we explicitly don't use ubuntu's image beacuse we don't want to use glean | 17:21 |
| clarkb | er | 17:21 |
| clarkb | cloud-init | 17:21 |
| fungi | clarkb: the pip issue may have been because it's a package provider and we may not have been able to say install system package before installing package resources requiring the pip provider | 17:21 |
| bkero | ok | 17:21 |
| AJaeger | pabelanger: 293767 LGTM | 17:21 |
| pabelanger | bkero: right, but fungi makes a good point. If you run the puppet module on another image without cron, you have the same failure. It seems sane to add ensure => package[cron], like you listed above | 17:22 |
| bkero | yup | 17:22 |
| EmilienM | oops | 17:22 |
| EmilienM | I think puppet beaker jobs are currently broken | 17:22 |
| EmilienM | http://logs.openstack.org/91/294091/2/check/gate-puppet-ceph-puppet-beaker-rspec-devstack-centos7/7217741/console.html#_2016-03-17_17_10_09_995 | 17:22 |
| *** nadya has quit IRC | 17:22 | |
| bkero | pabelanger: that's requiring a Package resource named 'cron'. So if there is none (like if cron is already installed), it will break | 17:22 |
| bkero | So you're inverting the broken :) | 17:22 |
| anteaya | EmilienM: yes, we are discussing the issue | 17:23 |
| pabelanger | bkero: yup | 17:23 |
| anteaya | EmilienM: merging this patch exposed that cron is not in the image when built | 17:23 |
| EmilienM | anteaya: thanks I did not follow it | 17:23 |
| anteaya | EmilienM: and openstackci jobs don't install cron though use it | 17:23 |
| anteaya | EmilienM: no worries, good eye | 17:24 |
| fungi | anteaya: EmilienM's error is on devstack-centos7 though | 17:24 |
| anteaya | oh sorry, I didn't look at the log | 17:24 |
| bkero | Yeah | 17:24 |
| EmilienM | I'm not sure we're talking about the same bug | 17:24 |
| bkero | `parse_hosts_file': Host file '/home/jenkins/workspace/gate-puppet-ceph-puppet-beaker-rspec-devstack-centos7/spec/acceptance/nodesets/nodepool-[centos7].yml' does not exist! (ArgumentError) | 17:24 |
| bkero | ^ That's different than cron | 17:24 |
| anteaya | EmilienM: fungi is correct, we don't seem to be, my apologies | 17:24 |
| EmilienM | anteaya: no worries at all | 17:24 |
| EmilienM | https://github.com/openstack/puppet-nova/blob/master/spec/acceptance/nodesets/centos-70-x64.yml | 17:25 |
| anteaya | so two breaks on puppet beaker jobs | 17:25 |
| clarkb | bkero: EmilienM that may acually be a jjb bug | 17:25 |
| anteaya | both from https://review.openstack.org/#/c/293719/ I wonder? | 17:25 |
| clarkb | bkero: EmilienM it should be 'centos7' but because its a list we get [centos7] ? | 17:25 |
| clarkb | thats a fun one | 17:25 |
| clarkb | anteaya: yes | 17:25 |
| fungi | yeah, i think that's to do with jjb embedding an explicit list construct | 17:25 |
| *** chem has joined #openstack-infra | 17:25 | |
| *** asettle has joined #openstack-infra | 17:25 | |
| EmilienM | fungi, clarkb: should we just wait jjb for updated? or? | 17:25 |
| bkero | Looks like one of the classic 2 problems of computer science | 17:25 |
| clarkb | EmilienM: we are going to reverthe change I think | 17:26 |
| anteaya | clarkb: I'm still in the revert camp | 17:26 |
| EmilienM | I hate reverts :( | 17:26 |
| fungi | so at this point, rolling back that job update sounds best, we have more than one issue to work through | 17:26 |
| anteaya | clarkb: do you want to offer? | 17:26 |
| *** jamesmcarthur has quit IRC | 17:26 | |
| clarkb | EmilienM: but that is a bug in jjb, when you give it multiple list entries it uses the strings but apparently a singleton list gives you a list | 17:26 |
| anteaya | fungi: agreed | 17:26 |
| EmilienM | clarkb: revert makes sense here, so many blockers | 17:26 |
| fungi | yeah, there's been an ongoing infra ml thread about that with jjb | 17:26 |
| fungi | the literal list issue i mean | 17:27 |
| *** degorenko is now known as _degorenko|afk | 17:27 | |
| *** jistr has quit IRC | 17:27 | |
| bkero | Shouldn't it be a simple naming fix in | 17:27 |
| bkero | jjb? | 17:27 |
| *** e0ne has quit IRC | 17:27 | |
| clarkb | bkero: sort of, its jjbs parsing of the yaml that is broken | 17:27 |
| clarkb | bkero: we can probably game that parsing to get something that works, not sure | 17:27 |
| clarkb | bkero: foo:\n - bar vs foo:\n bar | 17:28 |
| clarkb | we used the first, the second should work but they should be equivalent | 17:28 |
| *** mrmartin has joined #openstack-infra | 17:28 | |
| bkero | Ah | 17:28 |
| *** esikachev has quit IRC | 17:28 | |
| fungi | thread starts at http://lists.openstack.org/pipermail/openstack-infra/2016-March/003979.html | 17:28 |
| bkero | Sounds like hotfix to add '- ', then actually fix the jjb parsing some other time | 17:28 |
| *** moravec has joined #openstack-infra | 17:29 | |
| fungi | if you want to read up on the issue | 17:29 |
| *** Zhongjun_ is now known as Zhongjun | 17:29 | |
| clarkb | bkero: eaiser to just revert due to the cron thing | 17:29 |
| clarkb | according to http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html cron is part of posix | 17:29 |
| bkero | Ah, I didn't know they were the same ocmmit | 17:29 |
| clarkb | I think it is reasonable to have our base images provide a base system that attempts to provide posix utilities | 17:29 |
| fungi | and propose a revert of the revert noting need to confirm correct parsing of the job (by looking at the config check output) and also making sure we address the missing cron somehow before reapprovinf | 17:30 |
| clarkb | so in addition to the revert we should work on image updates to provide that base set of utilities (note uuidgen is not) | 17:30 |
| JayF | uuidgen ships with, of all things, e2fsutils, at least on some distros | 17:30 |
| anteaya | fungi: makes sense | 17:30 |
| JayF | e2fsprogs, I mean | 17:30 |
| *** SumitNaiksatam_ has quit IRC | 17:30 | |
| *** asettle has quit IRC | 17:31 | |
| *** vgridnev has quit IRC | 17:31 | |
| bkero | clarkb: that also includes qmail stuff, uucp, fort77, c99, and m4 | 17:31 |
| fungi | i think the argument for not including cron was an attempt to avoid shipping persistent daemons by default. though i think that's container-loving hippie mentality leaking into the implementation for our virtual machine images ;) | 17:31 |
| *** krtaylor has quit IRC | 17:31 | |
| anteaya | hippie's love containers by definition? | 17:32 |
| * anteaya updates her notes | 17:32 | |
| fungi | yes. it's a well-known fact | 17:32 |
| clarkb | bkero: ya it may not be perfect, but tools like df, du, xargs, cron, cp, getopts, etc are all very very reasonable to expect | 17:32 |
| anteaya | fungi: thanks for sharing | 17:32 |
| clarkb | kill locale printf sleep sort | 17:32 |
| pabelanger | clarkb: bkero don't forget ssh | 17:32 |
| bkero | clarkb: So what you're saying is... 'which subset of this list of posix utilities should we select' | 17:32 |
| clarkb | bkero: no, I am saying install the whole thing | 17:33 |
| fungi | bkero: i'm not sure where "qmail stuff" is coming from though. djbsoft is not anywhere in the posix spec | 17:33 |
| anteaya | can we get the revert in the process and then discuss the image fix? | 17:33 |
| bkero | including the fortran compiler? O_o | 17:33 |
| clarkb | bkero: ignore the ones that don't make a lot of sense because we get a bunch that do | 17:33 |
| *** akaszuba has quit IRC | 17:33 | |
| bkero | fungi: Sorry, there were a bunch of 'q' commands | 17:33 |
| anteaya | I think we have agreement revert is our happy path for the moment, don't we? | 17:33 |
| clarkb | hwhere ignore is install them and don't worry about it, not don't install them | 17:33 |
| pabelanger | okay, so has anybody proposed the revert? if not I can do that now | 17:33 |
| fungi | please do. i'll stop writing mine | 17:34 |
| bkero | clarkb: so 'install the subset of this list of posix utilities we select' and think about it as 'we want to install the whole thing, but we're only willing to go this far' | 17:34 |
| anteaya | pabelanger: I was waiting on clarkb, but no there isn't a revert up | 17:34 |
| pabelanger | ok | 17:34 |
| clarkb | bkero: no! | 17:34 |
| clarkb | bkero: install the whole thing | 17:34 |
| bkero | ok | 17:34 |
| clarkb | bkero: don't worry about the extra | 17:34 |
| clarkb | in that it will be there taking up a few megabytes of disk | 17:34 |
| *** yamamoto has joined #openstack-infra | 17:34 | |
| clarkb | compared to 17GB I don't think anyone should care | 17:34 |
| bkero | I think there's a significant portion of this list that aren't packaged too | 17:35 |
| bkero | Maybe they exist in some package | 17:35 |
| clarkb | ya so it may have to be a best effort | 17:35 |
| bkero | Like asa | 17:35 |
| clarkb | but we don't need to make arbitrary decisions. ubuntu does provide fort77 and its less than a megabyte according to aptitude | 17:36 |
| clarkb | ah its a wrapper so maybe larger depending on size of the thing wrapped | 17:36 |
| bkero | :) | 17:36 |
| bkero | probably a virtual | 17:36 |
| *** Guest57182 has quit IRC | 17:36 | |
| fungi | though i still think it's reasonable for puppet modules, job definitions, et cetera to be explicit about the packages they expect rather than telling people to go read the posix spec and come back when they've installed it all | 17:37 |
| bkero | 'eh, still not that bad: After this operation, 2,894 kB of additional disk space will be used. | 17:37 |
| *** cargonza has joined #openstack-infra | 17:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Move puppet jobs to ubuntu-trusty" https://review.openstack.org/294199 | 17:37 |
| pabelanger | change up | 17:37 |
| anteaya | fungi: and I think you make a very good point about openstackci especially, since it was created for external consumption | 17:37 |
| bkero | fungi: You're right, but this is how modules get bloated | 17:37 |
| *** vgridnev has joined #openstack-infra | 17:37 | |
| clarkb | fungi: yup, on the flip side though it seems reaonsable that every test job or shell script need not install cp or xargs or df or etc | 17:37 |
| *** jamesmcarthur has joined #openstack-infra | 17:37 | |
| clarkb | beacuse there is a reasonable expectation that a stnadard set of utilities exist on a machine | 17:38 |
| *** Sukhdev has joined #openstack-infra | 17:38 | |
| *** dprince has quit IRC | 17:38 | |
| anteaya | pabelanger: did you not just push the revert button on https://review.openstack.org/#/c/293719/ | 17:38 |
| anteaya | pabelanger: there is no comment there saying the patch is reverted | 17:38 |
| pabelanger | anteaya: I used git revert from commandline | 17:39 |
| fungi | yeah, if you use git revert it's helpful to go to the reverted change and leave a comment pointing to the rever | 17:39 |
| fungi | t | 17:39 |
| pabelanger | Yup, leaving comment now | 17:39 |
| bkero | SO if we are to include cron, which module should it live in, or should it live somewhere else as not a Puppet resource? | 17:39 |
| anteaya | pabelanger: ah okay, thanks | 17:40 |
| *** yamamoto has quit IRC | 17:40 | |
| clarkb | my preference would be that we have a dib element that we call ubuntu-useable or similar to install the things | 17:40 |
| pabelanger | anteaya: original patch updated | 17:40 |
| clarkb | sshd and uuidgen are two examples of things that should go in a "useable" image on top of what we might expect from the more minimal images | 17:40 |
| anteaya | pabelanger: thank you | 17:41 |
| fungi | pabelanger: if you can also make a revert of the revert, that would be awesome. set it as wip and add a comment about not approving until the missing cron package and jjb nested list issue are addressed | 17:41 |
| fungi | clarkb: i think greghaynes has already proposed that (called {distro}-packageinstall) though it's still got some -1's to work through | 17:42 |
| pabelanger | fungi: doing that now | 17:42 |
| *** smarcet has joined #openstack-infra | 17:42 | |
| *** krtaylor has joined #openstack-infra | 17:43 | |
| fungi | smarcet: jamesmcarthur: https://translate-dev.openstack.org/ logins are now going to openstackid-dev from what i can tell, sorry for the delay | 17:43 |
| clarkb | fungi: i will have to dig that up and see what the concerns are | 17:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Move puppet jobs to ubuntu-trusty"" https://review.openstack.org/294203 | 17:44 |
| jamesmcarthur | fungi: thanks for the update. we’ll start digging in | 17:44 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 17:44 |
| anteaya | pabelanger: how many jenkins need to be re-updated after your revert merges? | 17:45 |
| *** Sukhdev has quit IRC | 17:45 | |
| clarkb | fungi: https://review.openstack.org/#/c/211859/ that one? | 17:45 |
| *** fawadkhaliq has quit IRC | 17:46 | |
| pabelanger | anteaya: at least jenkins.o.o and jenkins05 for manual updates | 17:46 |
| pabelanger | that's the only 2 I did | 17:46 |
| zigo | fungi: clarkb: I tested my scripts here, and they seem to work: https://review.openstack.org/#/c/294022/ | 17:46 |
| fungi | clarkb: that's the one i saw, i think, yeah | 17:46 |
| anteaya | pabelanger: cool | 17:46 |
| pabelanger | but we'll have to confirm with others | 17:46 |
| zigo | One question though, should I edit the .gitrc with gitconfig, or it's done already by default? | 17:46 |
| *** tzn has quit IRC | 17:46 | |
| anteaya | pabelanger: and zuul? puppet will take care of that, yeah? | 17:46 |
| pabelanger | anteaya: hopefully | 17:47 |
| *** jamesmcarthur has quit IRC | 17:47 | |
| anteaya | great | 17:47 |
| zigo | sorry, .gitconfig, I mean. | 17:47 |
| *** kzaitsev_mb has quit IRC | 17:47 | |
| *** tzn has joined #openstack-infra | 17:47 | |
| greghaynes | clarkb: fungi Ya - so the concern is mostly 'what defines usable'. If there is something we can codify (like posix utilities) then thats easy - make an element which installs those things | 17:48 |
| *** fawadkhaliq has joined #openstack-infra | 17:48 | |
| *** matrohon has quit IRC | 17:48 | |
| pabelanger | zigo: I'd consider debian-jessie experimental atm, until we land a few infra JJB packages. That shouldn't stop you from your JJB changes, mostly a heads up | 17:49 |
| *** knikolla has joined #openstack-infra | 17:49 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 17:49 |
| zigo | pabelanger: Though my build scripts are tested and working... | 17:49 |
| krotscheck | So, it appears that we can no longer successfully capture a firefox instance in the gate. | 17:50 |
| * zigo goes out to fetch a pizza... | 17:50 | |
| krotscheck | Which is breaking every npm-based build that uses firefox. | 17:50 |
| krotscheck | Or wants to test against firefox. | 17:50 |
| *** dims has joined #openstack-infra | 17:50 | |
| *** nadya has joined #openstack-infra | 17:50 | |
| *** mrmartin has quit IRC | 17:50 | |
| clarkb | greghaynes: it looks like even openssh-server and ping are contentious | 17:50 |
| greghaynes | clarkb: yes | 17:51 |
| clarkb | greghaynes: how are people logging into their VMs? | 17:51 |
| anteaya | krotscheck: what do you mean capture a firefox instance? | 17:51 |
| fungi | clarkb: as i said, i think they're all making containers now | 17:51 |
| fungi | virtual machines are so last month | 17:51 |
| clarkb | I mean of all the tools including posix's funny list openssh-server seems like a straightforward thing to have | 17:51 |
| pabelanger | zigo: right, we just haven't launched a jenkins job yet | 17:52 |
| krotscheck | anteaya: To run browser tests, the gate spins up an instance of firefox in a virtual frame buffer, tells it to run tests, and then shut it down. The process of connecting the running firefox process with the testing harness is called capturing. | 17:52 |
| greghaynes | clarkb: yea.... I think the issue is mostly that folks dont want dib to become curator of a distro-like-thing which is an ubuntu package list for 'useful' | 17:52 |
| clarkb | greghaynes: But thats exactly what dibs prupose is... | 17:52 |
| anteaya | krotscheck: ah thank you | 17:52 |
| greghaynes | if theres an upstream thing we could point at thatd be awesome | 17:52 |
| clarkb | "make me an image" this requires opinions | 17:52 |
| *** tzn has quit IRC | 17:52 | |
| anteaya | krotscheck: could we ever capture firefox in the gate? | 17:52 |
| greghaynes | Yes, ideally dib isnt the one with those opinions though | 17:52 |
| clarkb | greghaynes: well all the cloud images built by distros I have used include sshd | 17:52 |
| krotscheck | anteaya: We could as of this morning | 17:52 |
| anteaya | krotscheck: is not being able to do so a new behaviour? | 17:52 |
| *** notnownikki has quit IRC | 17:52 | |
| krotscheck | It is a new behavior | 17:52 |
| krotscheck | http://logs.openstack.org/60/294160/3/check/gate-ironic-webclient-npm-run-test/0d3bd9c/console.html#_2016-03-17_17_49_29_605 | 17:52 |
| greghaynes | clarkb: Yep. So my thinking was to make -useful be cloud-image feature compatible | 17:53 |
| greghaynes | clarkb: and that would be an upstream opinion, not ours | 17:53 |
| clarkb | eh I think its fine to have that opinion | 17:53 |
| clarkb | we seem to be unable to have opinions in openstack | 17:53 |
| clarkb | despite having lots of exposure to things that make good opinions | 17:53 |
| clarkb | like qcow2 > vhd/raw for image uploads | 17:54 |
| anteaya | krotscheck: that is an example of a successful capture? | 17:54 |
| clarkb | and config drive > metadata server | 17:54 |
| greghaynes | clarkb: sure, so how would you codify the standard of what goes in and doesnt | 17:54 |
| smarcet | fungi: i was able to reproduce bug between translate dev and openstackid dev | 17:54 |
| greghaynes | If we can come up with something concise-ish then I would feel a lot better abou it | 17:54 |
| smarcet | fixing it :) | 17:54 |
| fungi | smarcet: that's excellent news! | 17:54 |
| fungi | thank you!!! | 17:54 |
| clarkb | greghaynes: I think you look at basic user needs. Users need remote access to instances. This requires an ssh daemon and a cloud init like utility | 17:54 |
| fungi | zigo: for our jobs which propose changes we call git config rather than writing to .git/config directly. similar in the git-review functional testsuite | 17:55 |
| krotscheck | anteaya: That's an example of an unscucessful capture | 17:55 |
| clarkb | greghaynes: now, of the options make an opinion on which is the most generic set. dropbear loses, openssh wins. cloud init probably wins, etc | 17:55 |
| krotscheck | anteaya: I have a confirmed successful run at 4AM this morning | 17:55 |
| greghaynes | clarkb: then then on top of that, all the posix tools? | 17:55 |
| krotscheck | Also running on firefox 45.0.0, so it's not the browser version. | 17:55 |
| clarkb | greghaynes: I like the idea of a separate posix element and that can be opt in or a dep | 17:55 |
| *** esikachev has joined #openstack-infra | 17:55 | |
| *** rcernin has quit IRC | 17:55 | |
| fungi | krotscheck: if those job logs include system and pip package lists we can diff them to see what has maybe updated | 17:56 |
| greghaynes | clarkb: Ya, I think that makes sense | 17:56 |
| *** tzn has joined #openstack-infra | 17:56 | |
| greghaynes | clarkb: So my thinking for the ssh/cloud-init like thing is that we can make element-deps a little bit smarter and facilitate that | 17:56 |
| *** tqtran-afk has joined #openstack-infra | 17:56 | |
| krotscheck | fungi: There shouldn't be any pip. Lemme see about system. | 17:56 |
| greghaynes | clarkb: where you can have a dependency that has a default, but if some other element provides that then dont use the default | 17:57 |
| krotscheck | fungi: http://logs.openstack.org/55/291755/7/check/gate-storyboard-webclient-npm-run-test-unit/86852d1/console.html is a successful run. | 17:57 |
| anteaya | krotscheck: ah okay thanks, sorry I couldn't interpret you log messages | 17:57 |
| clarkb | greghaynes: that makes sense. Then you can opt into glean over cloud init or dropbear over openssh if you choose | 17:57 |
| greghaynes | yep | 17:57 |
| *** tzn has quit IRC | 17:57 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:58 | |
| openstackgerrit | Nikolas Hermanns proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn jobs https://review.openstack.org/258522 | 17:58 |
| clarkb | krotscheck: that says chromium not firefox | 17:58 |
| *** SumitNaiksatam has quit IRC | 17:58 | |
| clarkb | oh its two lines one not part of the error, ignore me | 17:58 |
| *** tzn has joined #openstack-infra | 17:59 | |
| krotscheck | clarkb, fungi: looks like xvfb updated. | 17:59 |
| krotscheck | Which could cause the issue, if firefox doesn't know how to deal with that. | 17:59 |
| krotscheck | Wait, I lied | 18:00 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:00 | |
| *** doug-fish has quit IRC | 18:00 | |
| fungi | krotscheck: there's no explicit installed package list there, but we can extrapolate a bunch of it from the "Unpacking" lines | 18:00 |
| gsilvis | I'm looking to make a CI job for K2K federation. It'll need to spin up two devstacks, and do several operations to each of them. Where should I start looking for documentation on how to do this? | 18:00 |
| fungi | krotscheck: grep those out, sort them, then diff them | 18:00 |
| *** SumitNaiksatam has joined #openstack-infra | 18:00 | |
| *** doug-fish has joined #openstack-infra | 18:00 | |
| *** baoli_ has quit IRC | 18:01 | |
| *** hashar has joined #openstack-infra | 18:01 | |
| *** jordanP has quit IRC | 18:01 | |
| fungi | gsilvis: the openstack-infra/devstack-gate project has an ansible-based implementation for setting up devstack on two job workers with a tunneled network between them | 18:01 |
| *** jamesmcarthur has joined #openstack-infra | 18:02 | |
| fungi | gsilvis: there's also an ascii network diagram in there showing how they're connected | 18:02 |
| *** ddieterly[away] has quit IRC | 18:02 | |
| gsilvis | fungi: cool, I'm looking at that now | 18:03 |
| *** notnownikki has joined #openstack-infra | 18:03 | |
| *** HeOS has quit IRC | 18:03 | |
| fungi | gsilvis: presumably you want something like the current multi-node devstack jobs (used to test nova live migration, neutron dvr) but have it set up a keystone in each devstack with suitable configuration for what you want to model | 18:03 |
| clarkb | as an alternative you could probably just run two keystons | 18:04 |
| clarkb | not sure which of the two options are simpler | 18:04 |
| *** HeOS has joined #openstack-infra | 18:04 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests https://review.openstack.org/288840 | 18:04 |
| *** weshay has quit IRC | 18:04 | |
| fungi | oh, good point. may not actually need multi-node | 18:05 |
| clarkb | with multinode you don't have to juggle port configs on a single host, but you do have to juggle the networking | 18:05 |
| fungi | two keystone deployments on one server | 18:05 |
| clarkb | so there are tradeoffs for each option | 18:05 |
| *** doug-fish has quit IRC | 18:05 | |
| *** yamahata has joined #openstack-infra | 18:05 | |
| *** hashar has quit IRC | 18:06 | |
| *** jamesmcarthur has quit IRC | 18:06 | |
| gsilvis | clarkb: I think it'd be good to test actually doing openstack API calls across the two deployments, so I think I'll want to have more than just keystone | 18:06 |
| *** ddieterly has joined #openstack-infra | 18:07 | |
| *** gordc has quit IRC | 18:07 | |
| *** MarkAtwood has quit IRC | 18:08 | |
| *** tzn has quit IRC | 18:08 | |
| krotscheck | fungi: No version differences. | 18:09 |
| *** jamesmcarthur has joined #openstack-infra | 18:10 | |
| *** moravec has quit IRC | 18:10 | |
| *** ddieterly has quit IRC | 18:11 | |
| gsilvis | fungi: Could I have a link to one of the tests that is using this? I'm a little lost | 18:12 |
| *** piet has quit IRC | 18:13 | |
| *** tqtran-afk is now known as tqtran | 18:13 | |
| *** moravec has joined #openstack-infra | 18:14 | |
| pabelanger | clarkb: so, do you want to start with adding cron to our nodepool element first or wait until we get the changes you were talking about into diskimage-builder? | 18:14 |
| fungi | gsilvis: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n2001 the {pipeline}-tempest-dsvm-multinode-full{job-suffix} job-template | 18:14 |
| gsilvis | fungi: thanks, I'll take a look | 18:14 |
| *** jaosorior_away has quit IRC | 18:14 | |
| clarkb | pabelanger: I am waffling on that, I think our best bet for now is to make a small element in project-config/nodepool/elements that we put ssh server, uuidgen, cron, and any other things we can think of for the short term | 18:14 |
| *** nadya has quit IRC | 18:15 | |
| *** rbrndt has quit IRC | 18:15 | |
| clarkb | pabelanger: then when we can get a general solution in dib ew can switch elements and delete ours | 18:15 |
| fungi | gsilvis: it sets DEVSTACK_GATE_TOPOLOGY="multinode" but most of the logic in resides in devstack-gate and devstack | 18:15 |
| *** jaosorior_away has joined #openstack-infra | 18:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Move puppet jobs to ubuntu-trusty" https://review.openstack.org/294199 | 18:16 |
| clarkb | pabelanger: I can work on that element now if you aren't already | 18:16 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Bug Fixing https://review.openstack.org/294219 | 18:16 |
| gsilvis | fungi: yup, that makes sense | 18:17 |
| clarkb | pabelanger: the hard part is just building the pkg-map | 18:17 |
| *** angdraug has quit IRC | 18:17 | |
| *** pcaruana has quit IRC | 18:18 | |
| *** e0ne has joined #openstack-infra | 18:18 | |
| *** claudiub|2 has joined #openstack-infra | 18:19 | |
| greghaynes | clarkb: You could even make the posix thing in project-config and then copy it to dib if you like | 18:19 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make it easier to find 'add task' button https://review.openstack.org/294221 | 18:20 |
| greghaynes | clarkb: dib handles two elements with the same name in path just fine | 18:20 |
| fungi | that used to break it though, right? | 18:20 |
| greghaynes | nope | 18:20 |
| greghaynes | at least for as long as I remember | 18:20 |
| fungi | no, wait, it was two elements with the same filename in them | 18:20 |
| greghaynes | Yea, that shouldnt break, one overrides the other though | 18:20 |
| greghaynes | We did have a case where a downstream tool expected that to never happen, though | 18:21 |
| greghaynes | in tripleo-ci | 18:21 |
| fungi | because it tried to copy them all into the same location before running or something | 18:21 |
| AJaeger | pabelanger: | 18:21 |
| AJaeger | 293767 needs to be rebased now again ;( Sorry ;( | 18:21 |
| *** mrmartin has joined #openstack-infra | 18:21 | |
| *** claudiub has joined #openstack-infra | 18:22 | |
| pabelanger | clarkb: okay, I'll defer to you | 18:22 |
| greghaynes | You might be remembering some foot-gunning caused by that - if you name two files the same thing only one will ever get copied in (it gets copied over the other one) | 18:22 |
| AJaeger | pabelanger: let me rebase for you... | 18:22 |
| pabelanger | AJaeger: Ya, I think were holding off on JJB stuff so we can land 294203 | 18:22 |
| pabelanger | but that's fungi's call | 18:23 |
| pabelanger | I'm game either way | 18:23 |
| AJaeger | pabelanger: ok, then let's wait with rebase... | 18:23 |
| *** claudiub|2 has quit IRC | 18:24 | |
| *** hashar has joined #openstack-infra | 18:24 | |
| fungi | pabelanger: AJaeger: i think we ought to sort out the diskimage-builder and jenkins-job-builder pieces | 18:25 |
| *** moravec has quit IRC | 18:25 | |
| dmsimard | Are you guys aware of this error? Looks weird http://logs.openstack.org/91/294091/2/check/gate-puppet-ceph-puppet-beaker-rspec-ubuntu-trusty/88c17d9/console.html#_2016-03-17_17_10_07_179 | 18:26 |
| *** jamesmcarthur has quit IRC | 18:26 | |
| *** jaosorior_away has quit IRC | 18:26 | |
| clarkb | greghaynes: if you don't have a package in pkg-map then the name as given is used. What if I have a package in pkg-map and the current distro doesn't match any rules in pkg-map does it return nil or does it use the package name as given? | 18:26 |
| dmsimard | Oh wait, I totally misunderstood that error, not related to you guys | 18:26 |
| anteaya | dmsimard: yes, we have reverted that patch | 18:27 |
| anteaya | I'm not a guy | 18:27 |
| dmsimard | oh, thanks then anteaya | 18:27 |
| anteaya | it is related to a patch we merged a few hours ago | 18:27 |
| anteaya | we have reverted the patch | 18:27 |
| anteaya | pabelanger: is working on propigating the update | 18:27 |
| dmsimard | anteaya: didn't mean to discriminate with the "guys" :( | 18:27 |
| anteaya | hopefully we should be in a position where you can recheck in a few hours | 18:27 |
| dmsimard | anteaya: I know very well who you are! :p | 18:28 |
| anteaya | noone ever means to | 18:28 |
| anteaya | thank you | 18:28 |
| anteaya | but that is the current status on that issue in your logs | 18:28 |
| openstackgerrit | Merged openstack-infra/openstackid: Bug Fixing https://review.openstack.org/294219 | 18:28 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:28 | |
| dmsimard | anteaya: appreaciate it, thanks | 18:28 |
| pabelanger | ya, JJB should be starting updates now | 18:28 |
| AJaeger | fungi, ok. | 18:28 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 18:28 |
| anteaya | dmsimard: welcome | 18:28 |
| anteaya | pabelanger: awesome | 18:28 |
| pabelanger | so, what did we decide the fix to dmsimard log was? we need to update our beaker settings? | 18:29 |
| *** moravec has joined #openstack-infra | 18:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Bug Fixing https://review.openstack.org/294225 | 18:29 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 18:30 |
| *** pcaruana has joined #openstack-infra | 18:30 | |
| *** hashar has quit IRC | 18:30 | |
| clarkb | hrm doesn't appear to support groupinstalls either | 18:30 |
| fungi | pabelanger: or fix the yaml parser in jjb to not turn singleton lists into a literal list when embedding | 18:30 |
| lakshmiS | clarkb: Added a patch https://review.openstack.org/#/c/294224 for installing elasticsearch. i am definetly missing a lot but thought have to start somewhere and get some inputs on it | 18:31 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Bug Fixing https://review.openstack.org/294225 | 18:31 |
| *** sridhar_ram has quit IRC | 18:31 | |
| nibalizer | fungi: how critical is it that i send a gpg signed email if i've signed the tarball? | 18:31 |
| fungi | nibalizer: signing the tarball is probably file | 18:31 |
| fungi | fine | 18:31 |
| nibalizer | ok swee | 18:32 |
| nibalizer | then we're done in 10 seconds | 18:32 |
| lakshmiS | clarkb: let me know if i am completely on a wrong path or needs correction | 18:32 |
| fungi | nibalizer: i usually just attach the tarball and then use multi-part pgp/mime because that will sign the attachments too, but mutt makes that a lot easier than some muas | 18:32 |
| pabelanger | fungi: once JJB has been reverted for puppet jobs, would restarting zuul be on the table to pick up min-ready 1 and clear surplus nodes for devstack-trusty? | 18:32 |
| pabelanger | fungi: has anybody committed to that? | 18:32 |
| pabelanger | or EmilienM do you have a preference? | 18:33 |
| anteaya | nibalizer: nice work | 18:33 |
| fungi | pabelanger: well, it already picked up the min-ready update. it's just that we need a restart to clear out the old job registrations in gearman | 18:33 |
| pabelanger | fungi: Ah, okay. So, what sort things do we do when zuul is restarted? Any prep needed for it? | 18:34 |
| pabelanger | or just hold off until the queue is slow | 18:34 |
| *** piet has joined #openstack-infra | 18:34 | |
| nibalizer | fungi: yea i've just generated a detached signature | 18:35 |
| fungi | pabelanger: yeah, we export the check and gate pipelines immediately before restarting zuul, and then reenqueue them all. it can be a bit disruptive since it results in us dropping anything queued in other pipelines and also leaves a very brief window where zuul might miss some gerrit events too | 18:36 |
| *** dtantsur is now known as dtantsur|afk | 18:36 | |
| nibalizer | ok sent out | 18:36 |
| fungi | pabelanger: so doing it when zuul is less busy tends to be a little more comfortable | 18:36 |
| nibalizer | cc'd the infra list for funsies | 18:36 |
| fungi | nibalizer: thanks a bunch | 18:36 |
| nibalizer | ya no problem | 18:36 |
| nibalizer | sorry the turnaround was slow | 18:36 |
| fungi | i'll take a look and compare to my results and let you know if anything seems amiss | 18:37 |
| nibalizer | awesome ty | 18:37 |
| *** eharney has quit IRC | 18:37 | |
| nibalizer | im in EU timezone so I probably won't be too responsive from here on out | 18:37 |
| fungi | this still leaves the election officials ~5 hours to prep for sending ballots, so it's fine turnaround | 18:37 |
| nibalizer | until tomorrow I mean | 18:37 |
| openstackgerrit | Lakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight https://review.openstack.org/294224 | 18:37 |
| *** yamamoto has joined #openstack-infra | 18:38 | |
| *** Zhongjun has quit IRC | 18:38 | |
| *** rbrndt has joined #openstack-infra | 18:40 | |
| *** eharney has joined #openstack-infra | 18:41 | |
| greghaynes | clarkb: unsure about the missing distro case. I would hope it falls through | 18:41 |
| pabelanger | fungi: right, that makes sense. Do we have an ansible playbook to do that yet, or just shell commands currently? | 18:41 |
| greghaynes | clarkb: or, thats what id expect | 18:41 |
| *** ociuhandu has quit IRC | 18:41 | |
| *** tongli has quit IRC | 18:41 | |
| fungi | pabelanger: it's just some scripts run manually for now. we restart zuul pretty infrequently (or did up until the recent mass node changes which have left us needing a way to wipe out stale gearman job registrations) | 18:42 |
| *** jamesmcarthur has joined #openstack-infra | 18:42 | |
| anteaya | nibalizer: yeah thanks for that | 18:43 |
| pabelanger | I had a patch up to nodepool for a while, that did it automagically, but never did finish it off. | 18:43 |
| *** adrian_otto has quit IRC | 18:44 | |
| *** yamamoto has quit IRC | 18:44 | |
| clarkb | greghaynes: I am having a hard time running it locally since it apperas to infer element path magically? | 18:44 |
| *** gordc has joined #openstack-infra | 18:44 | |
| *** doug-fish has joined #openstack-infra | 18:45 | |
| *** doug-fish has quit IRC | 18:45 | |
| *** sambetts is now known as sambetts|afk | 18:45 | |
| * clarkb reads the source | 18:45 | |
| *** doug-fish has joined #openstack-infra | 18:45 | |
| *** baoli has joined #openstack-infra | 18:46 | |
| clarkb | oh it writes it out to /usr/share stuff | 18:47 |
| *** fitoduarte has joined #openstack-infra | 18:48 | |
| EmilienM | pabelanger: no pref | 18:48 |
| greghaynes | clarkb: unsure what 'it' is | 18:49 |
| *** doug-fish has quit IRC | 18:50 | |
| greghaynes | clarkb: oh, pkg-map? I'd just add a bash right after where it gets run and do a build | 18:50 |
| *** doug-fish has joined #openstack-infra | 18:51 | |
| clarkb | greghaynes: ya it being pkg-map | 18:52 |
| clarkb | its fine I managed to test what I need I think | 18:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add element to handle generic package installs https://review.openstack.org/294233 | 18:54 |
| clarkb | greghaynes: fungi pabelanger ^ thats a rough start | 18:54 |
| clarkb | I removed the stuff from the devstack element because we are using those nodes for so much more than devstack now | 18:55 |
| clarkb | and I think I did enough rpm -qf ing to figure out what is what on fedora/centos | 18:55 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages https://review.openstack.org/294022 | 18:56 |
| *** asettle has joined #openstack-infra | 18:57 | |
| *** jamesmcarthur has quit IRC | 18:57 | |
| pabelanger | clarkb: sure, I can test it out in a bit | 18:57 |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| clarkb | thta covers cron, uuidgen, rough build essential, and python things | 18:58 |
| anteaya | oops was going to review the elections specs and only 1 minute left | 18:59 |
| *** angdraug has joined #openstack-infra | 18:59 | |
| anteaya | fungi: can I convince you to wait for me? | 18:59 |
| anteaya | I won't let myself get distracted | 18:59 |
| anteaya | I've had them open for about 2 hours | 18:59 |
| *** sridhar_ram has joined #openstack-infra | 18:59 | |
| *** baoli has quit IRC | 19:00 | |
| *** asettle has quit IRC | 19:01 | |
| pabelanger | flushing the cache for JJB on jenkins.o.o, looks bad atm | 19:02 |
| *** baoli has joined #openstack-infra | 19:02 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Enable gate jobs for Packstack on the stable/mitaka branch https://review.openstack.org/294234 | 19:02 |
| *** pradk has quit IRC | 19:02 | |
| *** pvaneck has quit IRC | 19:03 | |
| *** esikachev has quit IRC | 19:03 | |
| clarkb | greghaynes: reading the code line 10 of https://review.openstack.org/#/c/294233/1/nodepool/elements/infra-package-needs/pkg-map should work | 19:03 |
| *** jcoufal has quit IRC | 19:03 | |
| clarkb | greghaynes: I tried to find a groupinstall option but couldnt find one | 19:03 |
| anteaya | fungi: done | 19:05 |
| anteaya | 5 after | 19:05 |
| fungi | anteaya: that's fine, i hadn't even started yet. still juggling a bunch of things | 19:06 |
| anteaya | yup figured | 19:06 |
| bkero | clarkb: so here's a package list. http://paste.openstack.org/show/490983/ | 19:06 |
| anteaya | just didn't want to miss, as I had lined them up long ago | 19:06 |
| *** vgridnev has quit IRC | 19:08 | |
| bkero | There's a bunch of dumb SCCS (Oracle's old version control bits) in there | 19:08 |
| *** krtaylor has quit IRC | 19:08 | |
| greghaynes | clarkb: yea, I don't think group install exists. Should be easy to add | 19:09 |
| *** abregman has quit IRC | 19:09 | |
| *** automagically has joined #openstack-infra | 19:11 | |
| clarkb | bkero: is that list for rhel or debian? | 19:11 |
| bkero | clarkb: ubuntu | 19:11 |
| *** eil397 has joined #openstack-infra | 19:11 | |
| *** kaisers_ has quit IRC | 19:11 | |
| *** ihrachys has quit IRC | 19:11 | |
| bkero | Most of it is shell built-ins or coreutils | 19:12 |
| clarkb | ya | 19:12 |
| smarcet | fungi: https://translate-dev.openstack.org/ is working fine | 19:12 |
| smarcet | fungi: i will prepare a new release for openstackid :) | 19:12 |
| fungi | smarcet: sounds great--thanks again. we'll get upgraded as quickly as possible and then can see whether that also takes care of refstack and the translation check site | 19:13 |
| *** fawadkhaliq has quit IRC | 19:13 | |
| bkero | clarkb: so I think the list is 'at bc binutils bison bsdmainutils bsdutils cflow coreutils cron cxref diffutils ed emacs-24-bin-common file findutils flex fort77 gawk gcc grep gzip libc-bin libc-dev-bin login m4 make man-db ncompress ncurses-bin patch procps sed sharutils sysvinit-utils talk util-linux uucp uustat vim' | 19:14 |
| clarkb | bkero: exuberant-ctags is what provides ctags according to my search | 19:15 |
| bkero | clarkb: that also works. I just did a dpkg -S $(which ctags) to find what the default has been | 19:16 |
| *** salv-orl_ has joined #openstack-infra | 19:16 | |
| bkero | They're both done through /etc/alternatives | 19:16 |
| *** vgridnev has joined #openstack-infra | 19:16 | |
| clarkb | bkero: did you want to propose a "posix-utils" element to dib? | 19:16 |
| clarkb | I can work on it if you don't have time | 19:16 |
| clarkb | bkero: gotcha | 19:16 |
| bkero | Yeah I can do that | 19:17 |
| greghaynes | hrm, that list seems like wed lose some valuable test coverage | 19:17 |
| greghaynes | gcc being a good example | 19:17 |
| *** pradk has joined #openstack-infra | 19:17 | |
| pabelanger | infra-root, looks like jenkins05 was the only JJB failure, looking into it now | 19:18 |
| clarkb | greghaynes: coverage of installs? | 19:18 |
| greghaynes | yes | 19:18 |
| anteaya | pabelanger: ack | 19:18 |
| greghaynes | I know I get nodes without gcc all the time | 19:18 |
| fungi | pabelanger: it probably is in need of a restart | 19:18 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenStackID 1.0.15 Release https://review.openstack.org/294242 | 19:18 |
| *** nmagnezi has quit IRC | 19:18 | |
| clarkb | greghaynes: infra already has build-essential listed | 19:18 |
| clarkb | greghaynes: so for us at least it doesn't make a difference | 19:19 |
| *** salv-orlando has quit IRC | 19:19 | |
| * bkero imagines gcc is part of build-essential, but isn't certain | 19:19 | |
| greghaynes | clarkb: I mean from the standpoint of 'devstack should work on a typical node' | 19:19 |
| smarcet | fungi: https://review.openstack.org/#/c/294242/ thx :) | 19:19 |
| clarkb | greghaynes: ya we already preinstall it | 19:19 |
| greghaynes | You do, I mean !infra users | 19:19 |
| clarkb | greghaynes: but isn't this an opt in? | 19:19 |
| clarkb | I guess I don't understand under what scenario we are losing test coverage | 19:20 |
| *** krtaylor has joined #openstack-infra | 19:20 | |
| greghaynes | I just mean, a user boots a cloud image and runs devstack. We dont verify that devstack installs build essential. That node will not have build-essential in it already | 19:20 |
| fungi | smarcet: yep, just finished reading the changelog. i'm going to update that commit message to link to the tag on git.openstack.org rather than on github.com, and then i'll approve it | 19:20 |
| *** Daisy has joined #openstack-infra | 19:20 | |
| *** baoli has quit IRC | 19:20 | |
| *** jamesmcarthur has quit IRC | 19:20 | |
| *** amitgandhinz has quit IRC | 19:20 | |
| clarkb | greghaynes: but we hvaen't ever covered that I don't think | 19:20 |
| *** baoli has joined #openstack-infra | 19:21 | |
| clarkb | so its not lost coverage | 19:21 |
| greghaynes | ah, ok. So then I would say - were better off if devstack installs that dep | 19:21 |
| *** hashar has joined #openstack-infra | 19:21 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: OpenStackID 1.0.15 Release https://review.openstack.org/294242 | 19:21 |
| *** moravec has quit IRC | 19:22 | |
| *** jamesmcarthur has joined #openstack-infra | 19:22 | |
| clarkb | I want to say we have needed it for our own bootstrapping | 19:22 |
| bkero | install gcc, bootstrap, remove gcc? | 19:23 |
| clarkb | not sure if that is still the case | 19:23 |
| greghaynes | :( | 19:23 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Enable gate jobs for Packstack on the stable/mitaka branch https://review.openstack.org/294234 | 19:23 |
| dmsimard | AJaeger: ^ couldn't have been long ago because until fairly recently we had stable branches going back to essex :p | 19:24 |
| *** e0ne has quit IRC | 19:24 | |
| dmsimard | we did a little spring cleaning | 19:24 |
| *** Daisy has quit IRC | 19:24 | |
| bswartz | does anyone know why wiki.openstack.org started requiring capcha solving to make changes? | 19:24 |
| dmsimard | I didn't want to explicitely exclude every branch :p | 19:25 |
| clarkb | bswartz: because it came under a massive spam attack | 19:25 |
| dmsimard | bswartz: to prevent spam | 19:25 |
| bswartz | oh! | 19:25 |
| bswartz | that sucks | 19:25 |
| anteaya | bswartz: indeed | 19:25 |
| fungi | bswartz: we also have new account creation temporarily disabled on it still | 19:25 |
| *** fverbs has quit IRC | 19:25 | |
| anteaya | wish folks had enough to entertain them that spamming would be eshewed | 19:25 |
| fungi | bswartz: because a lot of the spam was coming from valid launchpad openid accounts | 19:26 |
| *** yamahata has quit IRC | 19:26 | |
| bswartz | I would have assumed that openid would use captchas to prevent bogus accounts and offending accounts could simply be banned | 19:26 |
| pabelanger | fungi: you might be right about the restart of jenkins05, how do I check or proceed with that? | 19:26 |
| anteaya | bswartz: we assumed security that wasn't there | 19:26 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add element to handle generic package installs https://review.openstack.org/294233 | 19:27 |
| anteaya | and hasn't been implimented | 19:27 |
| clarkb | now with what should be valid json | 19:27 |
| anteaya | or I did, anyway | 19:27 |
| fungi | bswartz: yeah, the offending new accounts were coming in so fast that to get them all banned in a timely fashion we had to prevent new ones | 19:27 |
| anteaya | not sure if anyone else assumed anything about openid | 19:27 |
| greghaynes | ianw: http://logs.openstack.org/49/294149/1/check/gate-dib-dsvm-functests-ubuntu-trusty/be3ee5a/console.html | 19:27 |
| greghaynes | ianw: looks like dib gate is broken | 19:27 |
| clarkb | greghaynes: ya I think I got it fixed | 19:27 |
| greghaynes | ah | 19:27 |
| greghaynes | clarkb: what was it? | 19:28 |
| clarkb | greghaynes: try rerun and see if it works. ianw mostly fixed it but didn't source the venv activate script and its not executable so failed | 19:28 |
| bswartz | wasn't openstack going to roll its own ID at some point? is that canceled? deprioritized? | 19:28 |
| anteaya | bswartz: in progress | 19:28 |
| clarkb | greghaynes: pyyaml and uuidgen not installed on ubuntu-trusty | 19:28 |
| *** baoli has quit IRC | 19:28 | |
| clarkb | greghaynes: the fix was use a virtualenv to install dib but then that wasn't sourced properly | 19:28 |
| greghaynes | huh. | 19:28 |
| *** baoli has joined #openstack-infra | 19:28 | |
| fungi | bswartz: it's still a work in progress. we were considering switching the wiki to use it instead of launchpad, but someone needs to come up with a good user migration plan | 19:29 |
| clarkb | greghaynes: oh exceept now its fialing on something new? | 19:29 |
| greghaynes | clarkb: yea, this is an odd one | 19:29 |
| clarkb | PS1 unbound variable | 19:29 |
| clarkb | in the activate script | 19:29 |
| *** andymaier has quit IRC | 19:29 | |
| greghaynes | oh, were running with set -u | 19:30 |
| greghaynes | is probably the issue | 19:30 |
| clarkb | woo leaky bash options | 19:30 |
| clarkb | since its source ya | 19:30 |
| fungi | bswartz: it's actually already in use for several community systems (groups.o.o, refstack.o.o, translate.o.o, the translations check deployment) | 19:30 |
| *** mrmartin has quit IRC | 19:31 | |
| *** geekinutah has joined #openstack-infra | 19:31 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Disable set -u when sourcing dib venv https://review.openstack.org/294250 | 19:31 |
| *** kencjohnston_ has joined #openstack-infra | 19:32 | |
| *** apoorvad has quit IRC | 19:32 | |
| *** moravec has joined #openstack-infra | 19:32 | |
| *** Jeffrey4l_ has joined #openstack-infra | 19:33 | |
| *** Jeffrey4l has quit IRC | 19:34 | |
| *** cody-somerville has quit IRC | 19:35 | |
| *** shardy has quit IRC | 19:36 | |
| bswartz | I don't know about openid well enough to suggest a migration plan. Ideally systems like gerrit could allow both systems for a period of a few months and then you cut off the old one. publicize the hell out of it and pick a deadline that won't cause anyone to scream | 19:36 |
| *** davidlenwell has quit IRC | 19:36 | |
| *** apoorvad_ has joined #openstack-infra | 19:37 | |
| pabelanger | okay, jenkins05 seems to be the only master that isn't 100%. it will fail with 502 proxy error after some time | 19:37 |
| pabelanger | I am not sure how to fix that | 19:37 |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackID 1.0.15 Release https://review.openstack.org/294242 | 19:37 |
| pabelanger | all other masters appear to be reverted for puppet devstack-trusty | 19:37 |
| clarkb | bswartz: I think what we would attempt to do is update the gerrit db (or other services db) with openid urls for both systems like that, then remove the old urls after some time | 19:37 |
| clarkb | bswartz: the bigger issue is likely just figuring out how to do the mapping in the first place and keep it up to date | 19:38 |
| *** doug-fish has quit IRC | 19:38 | |
| *** ociuhandu has joined #openstack-infra | 19:38 | |
| *** vgridnev has quit IRC | 19:38 | |
| clarkb | pabelanger: go to https://jenkins05.openstack.org/monitoring and look at the thread count, if it is >1k ish then it is likely thread leaking | 19:38 |
| *** jamesmcarthur has quit IRC | 19:38 | |
| fungi | pabelanger: most likely suffering from the usual jenkins thread leak. there is a restart playbook but it tends to not work well once the server is misbehaving. first order of business is to get it into prepare for shutdown | 19:38 |
| *** nadya has joined #openstack-infra | 19:39 | |
| fungi | once you've confirmed via the method clarkb mentioned | 19:39 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Add experimental job for MongoDB https://review.openstack.org/294253 | 19:39 |
| flaper87 | IIRC, there's a way to run just 1 of the experimental jobs in a project. Did I dream this? | 19:39 |
| flaper87 | Like: check experimental $something | 19:39 |
| clarkb | flaper87: it was a dream | 19:40 |
| anteaya | tristanC tonyb what is happening with mordred's candidacy patch? https://review.openstack.org/#/c/292690/ | 19:40 |
| fungi | clarkb: bswartz: right, what we're lacking is a direct map between launchpad accounts and foundation member accounts. we'll need to come up with a best-effort map which takes into account that probably the only thing we can try to match between them is e-mail addresses, but they're not guaranteed to be the same for a given individual as people may have more than one e-mail address, and launchpad also | 19:40 |
| flaper87 | clarkb: ah, oh :( | 19:40 |
| fungi | defaults to making e-mail addresses private | 19:40 |
| flaper87 | clarkb: it was a nice dream, though | 19:40 |
| flaper87 | :P | 19:40 |
| *** jamesmcarthur has joined #openstack-infra | 19:41 | |
| pabelanger | clarkb: fungi: ya, jenkins05 not happy, currently still loading webui | 19:41 |
| anteaya | pabelanger: :( | 19:41 |
| tristanC | anteaya: well, if mordred doesn't abandon his candidacy, we'll have to approve it since it's valid | 19:42 |
| pabelanger | clarkb: okay, if reading this correct it is at 10821 threads? | 19:42 |
| *** davidlenwell has joined #openstack-infra | 19:42 | |
| fungi | pabelanger: it can take a few minutes and multiple retries sometimes when it gets this bad. if it gets so bad that doesn't pan out, the next possibility is that we ungracefully kill the service and deal with the aftermath | 19:42 |
| clarkb | pabelanger: nice! | 19:42 |
| *** peterlisak has quit IRC | 19:42 | |
| clarkb | pabelanger: definitely a leaking server so ya put it in shutdown mode (via the manage jenkins page), let the jobs finish then we clean it up and restart | 19:42 |
| anteaya | tristanC: I don't know of a provision in the rules of elections asking candidates to abandon their candidacy once posted | 19:42 |
| fungi | 11k is certainly > 1k | 19:43 |
| *** baoli has quit IRC | 19:43 | |
| pabelanger | okay, doing that now | 19:43 |
| tristanC | anteaya: if mordred abandon his candidacy, then packaging-deb will be ptl-less and it'll be up to the TC to appoint zigo (since we can't validate his candidacy) | 19:43 |
| fungi | anteaya: tristanC: the result can still be appealed to the tc though, and they can rule to invalidate afterward i think? | 19:43 |
| anteaya | well seems to be that mordred has expressed the desire to stand as candidate | 19:43 |
| *** onovy has quit IRC | 19:43 | |
| anteaya | if a candidate posts a candidacy and it is valid | 19:43 |
| pabelanger | clarkb: prepare for shutdown right? | 19:43 |
| *** baoli has joined #openstack-infra | 19:43 | |
| clarkb | pabelanger: yes | 19:44 |
| pabelanger | roger | 19:44 |
| tristanC | fungi: anteaya: well yes, either it's a tease/joke, then it could be abandone, otherwise we have to approve it | 19:44 |
| clarkb | pabelanger: what that does is stop any new jobs from starting but allows running jobs to finish without interruption | 19:44 |
| fungi | i can't recall whether the tc has the power to depose a ptl who is unwilling to abdicate, or if their only recourse is to make the program unofficial | 19:44 |
| pabelanger | right | 19:44 |
| fungi | s/program/project/ | 19:44 |
| anteaya | is mordred a valid candidatae or not | 19:44 |
| fungi | (we're changing terminology too often!) | 19:44 |
| tristanC | anteaya: mordred is a valid candidate by the book | 19:44 |
| anteaya | I dont' think the election officials have the place of decidig whether a candidacy is a joke | 19:45 |
| anteaya | just if it is valid | 19:45 |
| clarkb | https://review.openstack.org/#/c/294233/2 passes testing and is ready now if we want to try getting that in and rebuild some ubuntu-trusty images | 19:45 |
| clarkb | fungi: greghaynes pabelanger bkero ^ your feedback in particular would be valuable | 19:45 |
| anteaya | well my experience is that election officials follow the book | 19:45 |
| anteaya | which is my understanding of the role | 19:45 |
| pabelanger | looking | 19:45 |
| zigo | anteaya: Please don't waste your time with useless bureaucracy, nobody including myself cares about this debate, or the PLT badge, we only care for the project to be live and running. | 19:45 |
| zigo | anteaya: Likely, Monty himself doesn't care much. | 19:46 |
| anteaya | how do you know what monty cares about | 19:46 |
| anteaya | and I decide what I spend my time on | 19:46 |
| AJaeger | fungi, stackalytics has some mappings of email addresses already... | 19:46 |
| fungi | AJaeger: yes, it's one potential source we can draw from | 19:46 |
| anteaya | and there is a valid candidate that has announced his candidacy | 19:46 |
| zigo | anteaya: Ok, so let me say it in a less nice way, since you aren't being nice here: don't waste the time of others... | 19:46 |
| zigo | Thanks. | 19:46 |
| anteaya | nice doen'st enter into it | 19:47 |
| anteaya | a valid candidate has announced his candidacy | 19:47 |
| bkero | clarkb: cron might get grumpy if there's no sensible-editor installed | 19:47 |
| *** cody-somerville has joined #openstack-infra | 19:47 | |
| clarkb | bkero: let me see if the package deps handle that | 19:47 |
| bkero | I just tried in a chrooted one, and it got grumpy | 19:47 |
| pabelanger | clarkb: reviewed | 19:47 |
| lifeless | oh tox. tox tox tox | 19:47 |
| bkero | http://paste.openstack.org/show/490985/ | 19:47 |
| pabelanger | clarkb: so, is it now a hands off process for the jenkins restart or do we still need to manually trigger that? | 19:48 |
| lifeless | clarkb: mordred: you probably already know this, but tox presumes dependencies never change after a venv is made :( | 19:48 |
| zigo | anteaya: I don't see Monty in candidates/newton/Packaging-Deb | 19:48 |
| clarkb | pabelanger: you still need to do that when the jobs are all done | 19:48 |
| fungi | pabelanger: manually put it in prepare for shutdown, the ansible playbook isn't likely to have a lot of luck with it | 19:48 |
| clarkb | pabelanger: can walk you through that when it reaches that state | 19:48 |
| lifeless | http://tox.readthedocs.org/en/latest/_modules/tox/venv.html?highlight=nodeps <- | 19:48 |
| bswartz | clarkb fungi: what about a trasitional system that requires you to log in with your LP account to obtain a secret and then log in with your foundation account and provide that secret to prove you are the same person -- the system could then link the 2 IDs | 19:48 |
| *** piet has quit IRC | 19:48 | |
| anteaya | https://review.openstack.org/#/c/292690/ zigo | 19:48 |
| clarkb | lifeless: yes that is why we override the install commadn to pip install -u | 19:49 |
| anteaya | a vaild candidate has announced his candidacy | 19:49 |
| pabelanger | clarkb: fungi: okay, stepping away for a few minutes | 19:49 |
| fungi | bswartz: for some system, like the wiki, i think that will work though we need to do broad announcements and schedule the start and end of that transition | 19:49 |
| lifeless | clarkb: that doesn't fix it | 19:49 |
| zigo | LOL | 19:49 |
| lifeless | clarkb: it doesn't undo '--no-deps' | 19:49 |
| *** flepied has quit IRC | 19:49 | |
| fungi | bswartz: for others which don't allow the user to add multiple openid urls themselves, like gerrit, we'll need some tooling to handle that for us | 19:49 |
| *** flepied has joined #openstack-infra | 19:49 | |
| clarkb | bkero: could add vim-tiny | 19:49 |
| clarkb | let me see what the options are on centos | 19:50 |
| bswartz | gerrit keeps finding ways to screw us.... | 19:50 |
| bswartz | oh wait did I type that out loud? | 19:50 |
| fungi | bswartz: we can make it happen by adding them to the database but it will take admin intervention | 19:50 |
| clarkb | lifeless: overriding the install command should override the entire install command | 19:50 |
| clarkb | lifeless: so not sure where no-deps would be sneaking in from | 19:50 |
| clarkb | looks like fedora/centos is just "vim" as an option | 19:51 |
| *** baoli has quit IRC | 19:51 | |
| *** MarkAtwood has joined #openstack-infra | 19:52 | |
| lifeless | clarkb: from where I linked | 19:52 |
| lifeless | clarkb: see installpkg(...) | 19:52 |
| openstackgerrit | Merged openstack-infra/project-config: Disable set -u when sourcing dib venv https://review.openstack.org/294250 | 19:52 |
| tristanC | zigo: anteaya: alright, I hoped this packaging-deb PTL matter would get solved through discussion. Since zigo acknoledged the situation and mordred kept his candidacy open I'll approve the review now | 19:53 |
| *** doug-fish has joined #openstack-infra | 19:53 | |
| anteaya | tristanC: thank you | 19:53 |
| anteaya | I don't give a hoot who the ptl is | 19:53 |
| clarkb | lifeless: right I think we bypass all of that with our tox.inis | 19:53 |
| anteaya | but if a candidacy is valid it has to be approved | 19:53 |
| clarkb | lifeless: since it has to install the local package and we provide the entire command for it to do so | 19:54 |
| lifeless | clarkb: we don't - I just spent time with allison debugging a weird failure | 19:54 |
| lifeless | clarkb: with one of our stock tox.inis | 19:54 |
| clarkb | lifeless: which one? | 19:54 |
| tristanC | well at least, it seems like this debat bring some interrest back to supporting debian and package production through openstack-infra so that's a win! | 19:54 |
| anteaya | there is always a benefit to discussions, I agree | 19:54 |
| anteaya | thank you | 19:54 |
| lifeless | openstack-spec-tracker - I'll get you a url | 19:54 |
| *** _amrith_ is now known as amrith | 19:55 | |
| lifeless | clarkb https://github.com/allisonrandal/openstack-spec-tracker/blob/master/tox.ini | 19:55 |
| *** jamesmcarthur has quit IRC | 19:56 | |
| clarkb | ah I see its modifying opts with extraopts | 19:56 |
| clarkb | thats fail | 19:56 |
| lifeless | clarkb: yes | 19:56 |
| clarkb | so you could drop {opts} | 19:56 |
| clarkb | but that might break other things | 19:57 |
| lifeless | clarkb: right | 19:57 |
| pabelanger | and back | 19:57 |
| lifeless | clarkb: I'd like to get to the point of dropping -U | 19:57 |
| anteaya | pabelanger: welcome back | 19:57 |
| lifeless | clarkb: and I think I've gotten closer to the actual root cause of our headaches with this insight | 19:57 |
| lifeless | clarkb: wdyt? | 19:57 |
| clarkb | yes this would certianly explain some of the headache | 19:58 |
| *** Jeffrey4l_ has quit IRC | 19:58 | |
| lifeless | I'll add this to the trello I use for packaging important-things | 19:58 |
| lifeless | and make sure there is a bug | 19:58 |
| *** cp16net has left #openstack-infra | 19:59 | |
| *** jamesmcarthur has joined #openstack-infra | 20:00 | |
| *** Sukhdev has joined #openstack-infra | 20:01 | |
| *** pvaneck has joined #openstack-infra | 20:02 | |
| openstackgerrit | Tim Buckley proposed openstack-infra/system-config: Fix tab name indices in status template https://review.openstack.org/294266 | 20:02 |
| pabelanger | So, before I start diving down a rabbit hole, is there any reason that JJB does not use the C parser for PyYaml? In an effort to speed up things | 20:05 |
| anteaya | tristanC: in the Newton dir of the election repo there are two directories for refstack (Refstack and refstack) I'm sure that will keep happening, how do you want to address that when data is published according to the new spec? | 20:05 |
| clarkb | lifeless: seems like we might be able to get away with dropping {opts} if that helps | 20:06 |
| anteaya | pabelanger: sorry what are you currently addressing? | 20:06 |
| anteaya | pabelanger: I've lost track | 20:06 |
| clarkb | but I haven't tested that | 20:06 |
| lifeless | clarkb: I'd rather fix this, bump the tox minimunm and drop install_command | 20:06 |
| pabelanger | anteaya: was looking to see how to speed up JJB, since it takes a while to parse all our yaml files | 20:06 |
| *** fitoduarte has quit IRC | 20:06 | |
| tristanC | anteaya: that new directory creation slept through review process, I'm fixing it now | 20:06 |
| clarkb | pabelanger: probably only because the libyaml-dev headers are not present when installing pyyaml? | 20:06 |
| anteaya | tristanC: ah awesome, thanks | 20:07 |
| *** fitoduarte has joined #openstack-infra | 20:07 | |
| anteaya | pabelanger: oh okay so tuning not fixing a bug | 20:07 |
| anteaya | pabelanger: cool, sounds like a good idea | 20:07 |
| anteaya | pabelanger: and I don't have the answer to your question, sorry | 20:07 |
| pabelanger | clarkb: okay, so we've never attempted to use it I guess | 20:07 |
| *** doug-fish has quit IRC | 20:07 | |
| pabelanger | might hack on it tonight and see if there is a difference in time | 20:08 |
| fungi | smarcet: jamesmcarthur: it looks like slot1 (current active deployment) is running 1.0.15 and i'm still able to log into translate.o.o and refstack.o.o so i think we're in the clear for now! | 20:08 |
| *** pradk has quit IRC | 20:08 | |
| jamesmcarthur | fungi: great news! | 20:09 |
| fungi | thanks for working through this with us, and apologies for any delays this has caused on the summit apps end | 20:09 |
| clarkb | was the underlying issue a bug in openstackid or zanata? maybe both? | 20:09 |
| jamesmcarthur | fungi: perhaps we can discuss root access for that server for smarcet when things are a little less immediate :) I really think it would be a benefit for everyone as we could act more quickly to assist w/ all things openstackID. | 20:10 |
| clarkb | lifeless: ya I just have little hope for any contributions to tox as my attempts have in the past been largely ignored | 20:10 |
| openstackgerrit | Roman Podoliaka proposed openstack-infra/project-config: oslo.db: don't run tests with SQLAlchemy 0.9.x https://review.openstack.org/294122 | 20:11 |
| jamesmcarthur | clarkb: i believe it was a combo, but I’ve been frying other fish today. smarcet: can tell you for sure | 20:11 |
| lifeless | clarkb: Nakato has been having some success | 20:11 |
| ianw | clarkb / greghaynes: ah, yeah ... so is dib testing working now? | 20:12 |
| clarkb | ianw: maybe? we just had one more update to fix the set -u which hpefully is the last thing | 20:12 |
| jamesmcarthur | and fungi: no apologies necessary on the app front. we’re going to have to figure out a way to work through this one way or the other. Obviously we don’t want to disrupt active services for openstack users. At the same time, need to be able to do rapid development. I think having a better idea of dependencies is helpful for us, so we can be on the lookout and make sure we’re testing refstack and | 20:12 |
| jamesmcarthur | translate as well as www.o.o and groups.o.o | 20:12 |
| fungi | clarkb: the patch on the openstackid side was pretty subtle, fwiw. looks like mostly a logic error? https://review.openstack.org/294219 | 20:12 |
| greghaynes | ianw: unsure, just kicked off another recheck | 20:13 |
| *** doug-fish has joined #openstack-infra | 20:13 | |
| fungi | jamesmcarthur: yes, we had been under the impression we should be moving more services to openstackid as soon as time permits, but we could certainly do a better job of letting you know which ones they are (that is, when _we_ know about them anyway) | 20:13 |
| *** fitoduarte has quit IRC | 20:13 | |
| prometheanfire | clarkb: when was that merged? | 20:14 |
| clarkb | prometheanfire: 25 minute sago | 20:14 |
| prometheanfire | ok | 20:14 |
| ianw | greghaynes / clarkb : ok, cool, just coming online. change was untested but i think right direction. as long as we keep using "#!/usr/bin/env python" anyway ... | 20:14 |
| *** _ari_ has quit IRC | 20:14 | |
| prometheanfire | will push recheck button | 20:14 |
| jamesmcarthur | sure - i mean it’s an opensource openID project… so it’s great it’s getting used. | 20:14 |
| jamesmcarthur | we just didn’t realize it had been adopted by anyone but groups | 20:14 |
| jamesmcarthur | So I’m actually really excited about that part :) | 20:14 |
| prometheanfire | ianw: that should be python2 unless we know we are py3 compat | 20:15 |
| fungi | jamesmcarthur: also after the summit crunch i want do double down on getting the distro upgrades for openstackid-dev.o.o and openstackid.org done, and the openstackid-resources.o.o server deployed. it seems like some of the impact would be taking place over there in the future and would be less likely to disrupt authentication as a result, though i may be reading too much into the partitioning of | 20:16 |
| fungi | those apis | 20:16 |
| clarkb | prometheanfire: eh python says python == python2 | 20:16 |
| clarkb | prometheanfire: arch can continue to be broken if they choose | 20:16 |
| bkero | That's going to change in xenial too AIUI | 20:16 |
| greghaynes | bkero: nope | 20:17 |
| bkero | So ubuntu will be broken as well | 20:17 |
| bkero | No? | 20:17 |
| *** pcaruana has quit IRC | 20:17 | |
| greghaynes | bkero: just python2 isnt there by default AIUI | 20:17 |
| *** asettle has joined #openstack-infra | 20:17 | |
| clarkb | and no python | 20:17 |
| clarkb | only python3 | 20:17 |
| ianw | prometheanfire: yeah, at some point we were thinking of having tricks to handle that images might only ship python3 | 20:17 |
| fungi | bkero: i certainly hope no sane distros make /usr/bin/python run py3k | 20:17 |
| fungi | that's counter to what the python community recommends | 20:17 |
| ianw | i don't want to think about that today | 20:17 |
| prometheanfire | clarkb: gentoo does that too :P | 20:17 |
| fungi | archlinux was making a poor judgement call when they did that back in the 3.2 days | 20:18 |
| prometheanfire | clarkb: fedora defaults python to py3 now too iirc | 20:18 |
| clarkb | prometheanfire: well then gentoo can continue to be broken if they choose :P | 20:18 |
| prometheanfire | and fedora? | 20:18 |
| clarkb | prometheanfire: yes but fedora won't have a `python` | 20:18 |
| bkero | prometheanfire: but gentoo also has a sane system of alternatives | 20:18 |
| clarkb | only a `python3` | 20:18 |
| ianw | pabelanger: did you have a chance to see if centos-7 was uploading ok, but just timing out because we're taking too long to bring up network? | 20:18 |
| clarkb | default does not mean that the executable is called `python` it means this is what is installed by default | 20:18 |
| prometheanfire | oh ya, forgot fedora was doing it that way | 20:18 |
| clarkb | prometheanfire: that is the way python says you have to do it | 20:18 |
| prometheanfire | bkero: true | 20:18 |
| ianw | pabelanger: i have a vhd build with changed dependencies in glean that i will upload shortly and test out | 20:18 |
| prometheanfire | eselect python set python2 or something | 20:19 |
| pabelanger | ianw: no, I can check now | 20:19 |
| *** fawadkhaliq has joined #openstack-infra | 20:19 | |
| fungi | the python upstream recommendation is that you treat python and python3 as separate languages with separate interpreters. making your python3 interpreter answer to requests for python is all sorts of wrong | 20:19 |
| fungi | that's like deciding that your /bin/ksh should be a symlink to /bin/bash | 20:20 |
| prometheanfire | ya | 20:20 |
| clarkb | fungi: ah I see now looks like the function default is true so old code would've been fine with an else too, though new code is more readable so yay | 20:20 |
| bkero | fungi: or deciding /bin/vi should be a symlink to /bin/vim ;) | 20:20 |
| bkero | AIUI vim has a vi-emulation mode | 20:20 |
| clarkb | at least vim knows whta to do in that case | 20:20 |
| clarkb | bkero: ya it checks $0 | 20:21 |
| prometheanfire | how often do people go though upper-requirements to update? | 20:21 |
| ianw | fungi: it's a bit of a bummer though when you're trying to write something portable. istr there was a PEP, or maybe just talk, about an #! wrapper for python that autoselected | 20:21 |
| *** eharney has quit IRC | 20:21 | |
| prometheanfire | are we in lockdown now? | 20:21 |
| *** e0ne has joined #openstack-infra | 20:21 | |
| fungi | vim has a "vi" mode, python3 does not have a python (2.x) mode that it will engage when you don't call it as python3 | 20:21 |
| clarkb | prometheanfire: I believe requirements is locked down, but not sure if they are extending that to upper-constrinats | 20:21 |
| pabelanger | ianw: they look to be queued for upload | 20:21 |
| bkero | At this point is #!/usr/bin/env python more portable? or #!/usr/bin/env python2? | 20:21 |
| prometheanfire | my change is only to liberty | 20:21 |
| clarkb | bkero: they are supposed to be equivalent | 20:21 |
| *** kzaitsev_mb has joined #openstack-infra | 20:21 | |
| pabelanger | ianw: they've been inline for 2hours now | 20:21 |
| bkero | clarkb: AIUI some distros don't have a python2 binary/symlink | 20:22 |
| prometheanfire | been sitting for a week https://review.openstack.org/#/c/291859/ | 20:22 |
| *** asettle has quit IRC | 20:22 | |
| clarkb | bkero: right thats the other tricky thing | 20:22 |
| clarkb | bkero: and the other factor is some scripts iwll work in 2 and 3 | 20:22 |
| bkero | yup | 20:22 |
| pabelanger | ianw: currently 26 images that need to be uploaded still | 20:22 |
| mgagne | fungi: token is now 4h at internap. lets see if it changes something | 20:22 |
| clarkb | bkero: so for myself I have tried to stick to what python says is correct | 20:22 |
| clarkb | bkero: because at least then you can't point people that get it wrong at that | 20:22 |
| anteaya | I think I am going to head out for a walk in the sunshine | 20:23 |
| clarkb | fungi: https://review.openstack.org/#/c/294233/2 seems to be mostly happy | 20:23 |
| ianw | pabelanger: ok, well what about the centos-7 that has been uploaded? did sshing to that timeout? | 20:23 |
| fungi | mgagne: thank you! we're still debating what sane workarounds in shade would be for clouds with similarly short token lifetimes and no trusts support | 20:23 |
| clarkb | fungi: I can make a new ubuntu-trusty and fedora-23 if we can get that in | 20:23 |
| clarkb | mgagne: mordred seemd fine with gettin a new token before each upload too | 20:24 |
| mordred | yah | 20:24 |
| *** kgiusti has left #openstack-infra | 20:24 | |
| *** piet has joined #openstack-infra | 20:25 | |
| fungi | clarkb: i would have liked to see comments in package-installs.yaml warning about letting it become a dumping ground and contaminating our coverage of jobs installing the less common things they actually need, but that can always come in a later patch | 20:26 |
| *** jamesmcarthur has quit IRC | 20:26 | |
| pabelanger | ianw: no, they are still uploading | 20:26 |
| clarkb | bkero: re already installed things I figured it was good to overspecify that way if minimal becomes less minimal we are ok | 20:26 |
| fungi | (or more minimal!) | 20:26 |
| clarkb | bkero: and cron doesn't depend on an editor and I think puppet should work? | 20:26 |
| clarkb | fungi: er thats what I mean | 20:26 |
| AJaeger | prometheanfire: ask on #openstack-release for requirements | 20:27 |
| fungi | yeah, cron will work fine without an editor. crontab -e will not | 20:27 |
| clarkb | why don't we start there and if we need the editor can add that in too | 20:27 |
| ianw | pabelanger: but what about the image you shared with me yesterday? | 20:27 |
| *** aeng has joined #openstack-infra | 20:28 | |
| ianw | (btw you can probably unshare that, so it can be removed) | 20:28 |
| fungi | similarly, zigo did manage to download the debian-jessie qcow2 image so it can be deleted from /var/log/nodepool/images/ too | 20:28 |
| lifeless | Nakato: clarkb: https://bitbucket.org/hpk42/tox/issues/320/fails-to-detect-changed-dependencies-and <- please fix any mistakes I made :) | 20:29 |
| *** pcaruana has joined #openstack-infra | 20:30 | |
| pabelanger | ianw: I haven't done anything with it. Is it missing? | 20:31 |
| pabelanger | okay, I can unshare | 20:31 |
| pabelanger | and also remove debian-jessie | 20:31 |
| ianw | pabelanger: no ... i'm wondering why jobs still fail as NOT_REGISTERED if it's there, and it boots, and it comes online (eventually) | 20:31 |
| clarkb | lifeless: that sounds about right particularly the just let pip do its job line | 20:31 |
| ianw | pabelanger: all i can assume is that sshing to it times out before the network finishes timing out on dhcp and we *really* bring it up | 20:32 |
| ianw | but i'd like to confirm that a bit better before i go off trying to fix a problem that doesn't exist :) | 20:32 |
| *** jamesmcarthur has joined #openstack-infra | 20:33 | |
| clarkb | fungi: it may be worth restarting zuul today even without puppet nodes updated. THat will remove the vast number of registrations for devstack-trusty | 20:33 |
| fungi | clarkb: yes i agree, was just waiting to see activity (particularly in post) fall due to the backup we had earlier from repeatedly offlining the proposal slave | 20:34 |
| mordred | clarkb, fungi, mgagne: I'm going to try to hack on both the image import url and the re-get token patches on the plane this evneing | 20:34 |
| mordred | sorry I've been slammed in face-to-face meetings this week | 20:34 |
| pabelanger | ianw: right, new new images from yesterday. So, once they upload to rax, we can see if they reregister | 20:34 |
| pabelanger | clarkb: fungi: from what I see, JJB revert to puppet jobs is good | 20:35 |
| pabelanger | minus jenkins05, but we are fixing that now | 20:35 |
| fungi | pabelanger: thanks | 20:36 |
| pabelanger | okay, debian-jessie image removed from nodepool.o.o directory | 20:37 |
| fungi | pabelanger: the other thing i usually do while waiting for a jenkins master to go idle is delete any nodes from nodepool which are assigned to it and in a ready state, since they'll falsely fulfill demand even though they're not going to pick up any jobs | 20:37 |
| pabelanger | fungi: okay, I can start that now | 20:37 |
| fungi | that has a tendency to cause jobs to pile up waiting on new workers while nodepool doesn't realize it needs to build more | 20:37 |
| smarcet | fungi: openstackid 1.0.15 and both sites, refstack and zanata are working :) | 20:38 |
| openstackgerrit | Merged openstack-infra/project-config: Add element to handle generic package installs https://review.openstack.org/294233 | 20:38 |
| fungi | smarcet: yes, i agree! i checked on it earlier, and also gave the translation team a heads up that we'd upgraded again but that the regression should be gone now | 20:39 |
| *** matrohon has joined #openstack-infra | 20:39 | |
| fungi | smarcet: thanks for finding that and fixing, and for your patience while we did it through the dev environment | 20:39 |
| *** hamzy has quit IRC | 20:40 | |
| bookwar | hi, all, how to find jenkins log for documentation job? https://review.openstack.org/#/c/291457/ here for example, i'm trying to figure out why diagrams are not generated | 20:40 |
| bkero | clarkb: cron doesn't depend on an editor, but it tries to use sensible-editor when configuring | 20:40 |
| bkero | and since there's no editor installed, it fails | 20:40 |
| fungi | smarcet: from the fix, it looks like the underlying issue was a logic error introduced through an earlier refactor? | 20:41 |
| clarkb | bkero: when configuring the package install? | 20:41 |
| bkero | yup | 20:41 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job https://review.openstack.org/260466 | 20:41 |
| clarkb | bkero: wow, but shouldn't that be a dependency then? | 20:41 |
| bkero | clarkb: good question | 20:41 |
| clarkb | seems like a broken package :/ but we can add vim-tiny/vim to make it happy | 20:41 |
| smarcet | fungi: well issue was a log left from debugging , sorry , an stupid error | 20:41 |
| bkero | clarkb: here's me trying to install it: http://paste.openstack.org/show/490985/ | 20:41 |
| *** jamesmcarthur has quit IRC | 20:41 | |
| smarcet | seems that log line output to response and everything goes to hell | 20:42 |
| clarkb | bkero: does that return non zero | 20:42 |
| bkero | clarkb: ugh, nevermind. I thought sensible-editor was being used in package configure. Re-reading my pastebin it's just being used in crontab -e like one would expect | 20:42 |
| fungi | smarcet: oh, it was the log call you removed that had caused the authentication issue? that's even weirder | 20:42 |
| bkero | the failure was due to my chroot | 20:43 |
| bkero | and not having a proper /dev tree | 20:43 |
| clarkb | ah | 20:43 |
| AJaeger | bookwar: your docs gate has this URL http://docs-draft.openstack.org/57/291457/4/check/gate-fuel-docs-docs/e0c50a6//doc/build/html/ - replace docs-draft with logs and remove the /doc/build/html to get the log file | 20:44 |
| *** jamesmcarthur has joined #openstack-infra | 20:44 | |
| bookwar | AJaeger: cool, thank you | 20:45 |
| pabelanger | fungi: xargs to the rescue | 20:45 |
| AJaeger | bookwar: "build succeeded" - but I guess you did not generate content in the expected place which is "doc/build/html" | 20:45 |
| fungi | pabelanger: does nodepool delete take multiple node ids? i always assumed it would only use one | 20:45 |
| pabelanger | fungi: Ya, 1 at at time | 20:46 |
| AJaeger | bookwar: run "tox -e venv python setup.py build_sphinx" locally to verify | 20:46 |
| pabelanger | so did the following | 20:46 |
| pabelanger | sudo -H -u nodepool nodepool list | grep jenkins05 | grep ready | cut -d'|' -f 2 | xargs -n1 sudo -H -u nodepool nodepool delete | 20:46 |
| pabelanger | ugly, but worked | 20:46 |
| fungi | oh, -n1 | 20:46 |
| fungi | that's fancier than my ugly for loop, fwiw | 20:46 |
| pabelanger | I wish you could wildcard delete | 20:46 |
| smarcet | fungi: yes there were a log call like r_print(Input::all()) | 20:46 |
| smarcet | that was the culprit | 20:46 |
| fungi | smarcet: yep, saw it. that's a devious bug--good find | 20:47 |
| fungi | pabelanger: well, you can. it's called an sql query | 20:47 |
| fungi | but that gets tricky with the subnode table | 20:47 |
| pabelanger | fungi: I too like to live dangerously | 20:48 |
| bookwar | AJaeger: i choose the wrong example, the path issue is already solved by another patch, but my problem is this one http://docs-draft.openstack.org/03/292403/3/check/gate-fuel-docs-docs/4be4048//doc/build/html/devdocs/develop/architecture.html - the ".. uml:" is not parsed into diagram, as it is expected https://docs.fuel-infra.org/fuel-dev/develop/architecture.html | 20:48 |
| bookwar | is plantuml for docs supported in gate somehow? | 20:49 |
| *** weshay has joined #openstack-infra | 20:50 | |
| pabelanger | fungi: clarkb: I see 3 jobs in the build queue on jenkins05, do I manually cancel them in the UI? | 20:51 |
| clarkb | pabelanger: no you can ignore those | 20:51 |
| fungi | pabelanger: i usually just ignore them. i'm not sure how they get there or what happens to them honestly | 20:51 |
| *** jamesmcarthur has quit IRC | 20:51 | |
| clarkb | its a gearman plugin thing I think | 20:51 |
| pabelanger | Ya, one queued for 15 days | 20:51 |
| AJaeger | bookwar: platuml at least installs fine in the job according to the log file | 20:52 |
| *** ZZelle_ has joined #openstack-infra | 20:52 | |
| *** cj has quit IRC | 20:53 | |
| AJaeger | bookwar: but I've never used it, so no idea what dependencies it has | 20:53 |
| bookwar | AJaeger: i see, i'll try to dig in a bit.. | 20:53 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/pbr: Include wsgi_scripts in generated wheels https://review.openstack.org/280270 | 20:53 |
| AJaeger | bookwar: if you figure out what's missing, you can add thoese locally for your repo. | 20:53 |
| AJaeger | An other-requirements.txt file can be used but that's still a bit experimental. Best talk with others here since I'm signing off for now... | 20:54 |
| * AJaeger waes good-bye | 20:54 | |
| *** cj has joined #openstack-infra | 20:55 | |
| *** aysyd has quit IRC | 20:55 | |
| *** MarkAtwood has quit IRC | 20:56 | |
| *** ihrachys has joined #openstack-infra | 20:58 | |
| *** jamesmcarthur has joined #openstack-infra | 20:58 | |
| *** fawadkhaliq has quit IRC | 21:00 | |
| *** kaisers_ has joined #openstack-infra | 21:00 | |
| *** moravec has quit IRC | 21:00 | |
| *** moravec has joined #openstack-infra | 21:02 | |
| *** jamesmcarthur has quit IRC | 21:02 | |
| *** akshai_ has joined #openstack-infra | 21:03 | |
| *** rossella_s has quit IRC | 21:03 | |
| *** rossella_s has joined #openstack-infra | 21:03 | |
| pabelanger | clarkb: okay, jobs done on jenkins05, but still a few nodes listed as offline | 21:04 |
| *** e0ne has quit IRC | 21:04 | |
| fungi | nibalizer: tristanC: i've finished looking for the electorate tarball e-mail and since determined that mailman probably rejected the post. hence only tristanC's reply went to the ml | 21:04 |
| *** akshai has quit IRC | 21:05 | |
| *** kaisers_ has quit IRC | 21:05 | |
| clarkb | pabelanger: you can manually delete them in the ui or if its terribly slow between stopping and starting jenkins service edit the config.xml to remove all the slaves and set nodepool to delete the ones on 05 it knows about | 21:05 |
| *** |-paul-| has joined #openstack-infra | 21:06 | |
| *** rfolco has quit IRC | 21:06 | |
| *** sdague has quit IRC | 21:06 | |
| prometheanfire | clarkb: dib still fails | 21:06 |
| *** asettle has joined #openstack-infra | 21:06 | |
| prometheanfire | 2016-03-17 20:33:12.867 | ./env/bin/activate: line 13: _OLD_VIRTUAL_PYTHONHOME: unbound variable | 21:06 |
| prometheanfire | centos one | 21:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove some extra mitaka translation jobs https://review.openstack.org/294286 | 21:06 |
| prometheanfire | http://logs.openstack.org/94/290894/3/check/gate-dib-dsvm-functests-devstack-centos7/e4744ee/console.html | 21:06 |
| clarkb | prometheanfire: I think that just means jjb has't updated the jobs yet | 21:08 |
| *** ihrachys has quit IRC | 21:08 | |
| prometheanfire | so wait til then I guess | 21:08 |
| *** kencjohnston_ has quit IRC | 21:08 | |
| ianw | prometheanfire: yep -> http://logs.openstack.org/94/290894/3/check/gate-dib-dsvm-functests-devstack-centos7/e4744ee/console.html#_2016-03-17_20_16_24_357 | 21:09 |
| pabelanger | clarkb: okay, slaves deleted via the UI | 21:09 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add non voting job to test k8s with dib atomic image https://review.openstack.org/294112 | 21:09 |
| *** ihrachys has joined #openstack-infra | 21:09 | |
| prometheanfire | well, recheck in 30 min | 21:10 |
| *** asettle has quit IRC | 21:11 | |
| *** Sukhdev has quit IRC | 21:12 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting https://review.openstack.org/292894 | 21:12 |
| *** ldnunes has quit IRC | 21:12 | |
| *** kencjohnston_ has joined #openstack-infra | 21:13 | |
| *** jamesmcarthur has joined #openstack-infra | 21:13 | |
| *** moravec has quit IRC | 21:14 | |
| fungi | pabelanger: clarkb: any more i've just gotten used to editing the config.xml and deleting all lines between <slaves> and </slaves> before starting the service back up | 21:15 |
| *** nadya has quit IRC | 21:15 | |
| *** regXboi has quit IRC | 21:15 | |
| *** fawadkhaliq has joined #openstack-infra | 21:15 | |
| fungi | thanks for the resend, nibalizer! | 21:16 |
| pabelanger | fungi: clarkb: So, I should have jenkins stopped now and do that step? | 21:16 |
| *** jamesmcarthur has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack/requirements: uc: openstackdocstheme==1.3.0 https://review.openstack.org/293859 | 21:16 |
| clarkb | pabelanger: yup | 21:17 |
| clarkb | pabelanger: just remove everything between the two slaves tags | 21:17 |
| fungi | pabelanger: once all jobs have finished, you need to `sudo service jenkins stop` and then keep an eye on the process table for the java processes to die off | 21:17 |
| clarkb | oh ya it may not die and need kill -9ing | 21:17 |
| clarkb | (which is in the playbook fwiw) | 21:17 |
| fungi | and sometimes they don't, and you need to get increasingly aggressive with kill signals | 21:17 |
| crinkle | fungi: clarkb: jeblair: yolanda: rcarrillocruz: greghaynes: SpamapS: Venu and wendar would like to set up a call to review our infra-cloud networking request and discuss a network diagram, would some of you wish to attend? (also posted to the infra ml) | 21:17 |
| fungi | right that | 21:17 |
| fungi | crinkle: saw the e-mail, thanks! | 21:18 |
| fungi | crinkle: what timeframe are we looking at for the call? just sometime next week (tbd)? | 21:18 |
| rcarrillocruz | crinkle: i'll check the time, sure | 21:18 |
| *** moravec has joined #openstack-infra | 21:18 | |
| fungi | crinkle: nevermind, i see you said Tuesday at 1800 UTC' | 21:19 |
| crinkle | fungi: sometime next week is the time frame, I'm hoping we can come up with a couple of proposals to find something that works for us and Venu and wendar | 21:19 |
| rcarrillocruz | tuesday works for me ( i think ) | 21:19 |
| fungi | crinkle: the proposed time wfm | 21:19 |
| rcarrillocruz | i'll respond in mailing list | 21:19 |
| fungi | it's right before the infra team meeting, but as long as it's under an hour should be no problem | 21:19 |
| *** knikolla has quit IRC | 21:20 | |
| fungi | nibalizer: tristanC: so far the only differences i see between the electorate provided and the version i ran yesterday is a bunch of people who updated their preferred e-mail addresses in gerrit at the last possible moment, but that's not unusual. looks good! | 21:23 |
| fungi | tonyb: ^ | 21:23 |
| pabelanger | okay, cleared up and starting jenkins05 again | 21:23 |
| *** baoli has joined #openstack-infra | 21:23 | |
| *** jamesmcarthur has joined #openstack-infra | 21:23 | |
| tonyb | fungi: Thanks. good to know. | 21:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Add Tempest check runs graph https://review.openstack.org/294293 | 21:24 |
| fungi | counts and user ids didn't change, which is the important bit | 21:24 |
| pabelanger | and looks to be launching jobs | 21:25 |
| pabelanger | cool | 21:25 |
| pabelanger | clarkb: I'll trying the playbook next time | 21:25 |
| *** gordc has quit IRC | 21:25 | |
| *** ihrachys_ has joined #openstack-infra | 21:26 | |
| *** denisra has joined #openstack-infra | 21:26 | |
| fungi | tristanC: tonyb: nibalizer: looks like the extra step of copying the stable branch electorate in from the other run didn't happen, but that's probably fine for now since nobody stepped up to be ptl for them? | 21:27 |
| *** matrohon has quit IRC | 21:27 | |
| tonyb | fungi: that's my understanding. I think it will fall on the TC to make a decision. | 21:28 |
| fungi | okay, then you should have what you need to move forward with the contested positions | 21:28 |
| *** ihrachys has quit IRC | 21:28 | |
| fungi | but let us know if you need anything | 21:28 |
| *** sdake has quit IRC | 21:28 | |
| jeblair | crinkle: that time sounds great | 21:29 |
| *** amotoki has quit IRC | 21:29 | |
| tonyb | lots of actual elections this time round. | 21:29 |
| *** geekinutah has quit IRC | 21:30 | |
| *** smarcet has quit IRC | 21:30 | |
| *** sdake has joined #openstack-infra | 21:30 | |
| *** claudiub has quit IRC | 21:31 | |
| *** jamesmcarthur has quit IRC | 21:31 | |
| tonyb | fungi: on a tangental note what happen now with the electiom infra-specs? | 21:31 |
| fungi | tonyb: makes me glad we're not on the old script that required hand-assembling the team electorates out of per-repo lists, adding extra-atcs in by hand, manually searching for duplicate accounts... | 21:31 |
| *** matrohon has joined #openstack-infra | 21:31 | |
| fungi | tonyb: i am now 2.5 hours past due for approving the two patches for that | 21:31 |
| fungi | thanks for the reminder ;) | 21:31 |
| tonyb | fungi: it wasn't a nagg it's just my first time throught tht process | 21:32 |
| fungi | tonyb: after the day we've had in here, can use all the reminder help i can get | 21:32 |
| *** jamesmcarthur has joined #openstack-infra | 21:32 | |
| *** vgridnev has joined #openstack-infra | 21:33 | |
| tonyb | fungi: As I'm just gettign started I don't knwo what happend but still I has a sad if y'all had a rough time | 21:33 |
| *** jamesmcarthur has quit IRC | 21:33 | |
| *** lakshmiS has quit IRC | 21:34 | |
| *** vgridnev has quit IRC | 21:34 | |
| fungi | tonyb: it's just been a busy day. omelettes and broken eggs, you know | 21:34 |
| *** doug-fish has quit IRC | 21:35 | |
| *** jamesmcarthur has joined #openstack-infra | 21:36 | |
| *** Daisy has joined #openstack-infra | 21:36 | |
| anteaya | tonyb: so basically after fungi hits go you work on the bits you said you would | 21:36 |
| anteaya | and then tell us so we can do the things we said we would do, like review | 21:37 |
| fungi | go has already been hit, it'll just be a few minutes until these merge | 21:37 |
| tonyb | anteaya: jhesketh has done them all for me :) | 21:37 |
| anteaya | isn't he wonderful | 21:37 |
| fungi | and yeah, jhesketh wrote a preliminary implementation | 21:37 |
| tonyb | anteaya: https://review.openstack.org/#/q/topic:add_elections yes he is | 21:37 |
| anteaya | everyone should have a jhesketh in their life | 21:37 |
| fungi | predicting approval of the current spec changes | 21:37 |
| fungi | accurately predicting, i might add | 21:37 |
| anteaya | tonyb: then you bug us til they merge | 21:37 |
| *** moravec has quit IRC | 21:38 | |
| anteaya | ha ha ha | 21:38 |
| anteaya | crystal ball he has | 21:38 |
| openstackgerrit | Merged openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 21:38 |
| openstackgerrit | Merged openstack-infra/infra-specs: Mark public hiera as completed https://review.openstack.org/285753 | 21:38 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Reduced log level from warning to error on global config https://review.openstack.org/294296 | 21:38 |
| *** maratoid has joined #openstack-infra | 21:38 | |
| *** ihrachys_ has quit IRC | 21:39 | |
| maratoid | Hello. Is this the right place to ask questions about jenkins job builder? | 21:39 |
| anteaya | yes | 21:39 |
| maratoid | Thanks! | 21:39 |
| *** ihrachys has joined #openstack-infra | 21:39 | |
| anteaya | welcome | 21:40 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Reduced log level from warning to error on global config https://review.openstack.org/294296 | 21:40 |
| anteaya | wish they all were that easy | 21:40 |
| maratoid | I can't quite get past this error: "Duplicate entry point found for component type: 'property', 'property',name: 'build-discarder'" | 21:40 |
| anteaya | maratoid: we like stacktraces | 21:40 |
| *** doug-fis_ has joined #openstack-infra | 21:40 | |
| maratoid | i see. | 21:40 |
| anteaya | posted to a paste service is our favourite | 21:40 |
| *** Daisy has quit IRC | 21:40 | |
| *** jamesmcarthur has quit IRC | 21:40 | |
| anteaya | paste.openstack.org unless you prefer something else | 21:41 |
| maratoid | http://paste.openstack.org/show/491000/ | 21:41 |
| *** mriedem has quit IRC | 21:41 | |
| anteaya | quick you are | 21:42 |
| fungi | maratoid: is this a regression from a recent jenkins-job-builder upgrade, or new configuration you're trying to work through? | 21:42 |
| *** thorst has quit IRC | 21:42 | |
| maratoid | new configuration | 21:42 |
| jhesketh | Morning | 21:43 |
| anteaya | morning jhesketh | 21:43 |
| *** thorst has joined #openstack-infra | 21:43 | |
| fungi | easiest way to debug these things is to distill your config down to a minimal version which reproduces the error | 21:43 |
| *** jamesmcarthur has joined #openstack-infra | 21:43 | |
| fungi | and then provide a paste of that minimal configuration so that we can more easily spot the issue in it | 21:44 |
| fungi | often times i find the answer myself while i'm trying to come up with a minimal reproduction | 21:44 |
| *** sdake has quit IRC | 21:45 | |
| *** doug-fis_ has quit IRC | 21:45 | |
| maratoid | seems like this comes up for me whenever i have more than 1 job template in my project, using build-discarder | 21:45 |
| *** doug-fish has joined #openstack-infra | 21:45 | |
| *** jpr has quit IRC | 21:45 | |
| maratoid | perhaps i misunderstood how job templates work. do i need to make sure they are in a job group, or anything like that? | 21:46 |
| *** fawadkhaliq has quit IRC | 21:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Boot from volume in ping test https://review.openstack.org/293512 | 21:46 |
| *** thorst has quit IRC | 21:47 | |
| *** xyang1 has quit IRC | 21:47 | |
| JayF | I'm trying to track down a gate failure building IPA ( http://logs.openstack.org/24/160224/23/check/gate-tempest-dsvm-ironic-pxe_ipa-src/64c1f18/logs/devstacklog.txt.gz#_2016-03-17_21_09_26_129 ) -- is there any way to get onto one of these machines when it fails before it's deleted? | 21:48 |
| *** tonytan4ever has quit IRC | 21:48 | |
| anteaya | maratoid: can you paste what you have for templates? | 21:48 |
| JayF | Or a script I can run in a VM / cloud server to reproduce the build exactly like it happens in the devstack gate | 21:48 |
| *** baoli has quit IRC | 21:49 | |
| *** doug-fish has quit IRC | 21:50 | |
| *** baoli has joined #openstack-infra | 21:50 | |
| clarkb | JayF: there should be a reproduce.sh script in the logs/ dir | 21:50 |
| JayF | oooh | 21:50 |
| clarkb | JayF: now that requires you provide your own image/OS but its a start | 21:51 |
| JayF | Just a trusty VM, right? | 21:51 |
| maratoid | anteaya: https://github.com/maratoid/kraken-ci-jobs + http://paste.openstack.org/show/491005/ | 21:51 |
| clarkb | if you can't reproduce with that then we can hold an instance for you | 21:51 |
| clarkb | JayF: yes | 21:51 |
| JayF | Cool, I have a cluster of big beefy servers just waiting to run this :P | 21:51 |
| maratoid | anteaya: I'll try to distill to a smaller example | 21:51 |
| JayF | NobodyCam: ^ going to start that science, but I'm out Fri/Mon so hopefully 2 hours is enough :P | 21:51 |
| NobodyCam | JayF: please let me know how far you get and what you had to do :) | 21:53 |
| NobodyCam | and Morning / afternoon / evening Infra peeps :) | 21:53 |
| JayF | NobodyCam: I'm doing this on an OnMetal(tm) v2 server, so hopefully it won't take long :P | 21:54 |
| NobodyCam | nice :) | 21:54 |
| clarkb | JayF: based on my reading of the log looks like there is no docker socket to connect to, maybe docker isn't autostarting? | 21:55 |
| clarkb | or permissions have hcanged/ | 21:55 |
| JayF | clarkb: docker is starting, I think perms are likely | 21:55 |
| *** asettle has joined #openstack-infra | 21:55 | |
| anteaya | maratoid: yeah it is going to take me some time to figure out how your files are structured | 21:55 |
| JayF | clarkb: initial thought was the socket wasn't getting fully created, so we added a busyloop to check for the socket, that passes, and it still fails | 21:55 |
| anteaya | maratoid: I think simplified would save both of us time | 21:55 |
| JayF | clarkb: so I can keep guessing and eating up gate resources or I can repro and find which magic fix it is :P | 21:55 |
| JayF | clarkb: https://review.openstack.org/#/c/294172/ being our first stab at it | 21:56 |
| *** doug-fish has joined #openstack-infra | 21:56 | |
| JayF | clarkb: (you won't see the failure there, because we don't run ipa -src jobs against Ironic, but we have it testing against another commit in IPA with depends-on) | 21:56 |
| anteaya | clarkb: should firefox be forcing me to download reproduce.sh from http://logs.openstack.org/24/160224/23/check/gate-tempest-dsvm-ironic-pxe_ipa-src/64c1f18/logs/ when I click on it? | 21:57 |
| *** kencjohnston_ has quit IRC | 21:57 | |
| clarkb | anteaya: it isn't treated as a text file due to the suffix so yes it will want you to download | 21:57 |
| *** fawadkhaliq has joined #openstack-infra | 21:57 | |
| anteaya | chrome doesn't | 21:57 |
| clarkb | which makes sense, you have to execute it, viewing it in the browser isn't helpful | 21:57 |
| clarkb | anteaya: chrome may be smart then | 21:57 |
| anteaya | seems to be | 21:58 |
| anteaya | but thanks, didn't know about reproduce.sh | 21:58 |
| clarkb | fyi I have filed https://bugs.launchpad.net/cinder/+bug/1558807 and have left my test instance and volume up in vexxhost so that it can be poked at more as part of bug triage if necessary | 21:58 |
| openstack | Launchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,New] | 21:58 |
| anteaya | clarkb: ack | 21:59 |
| *** jamesmcarthur has quit IRC | 21:59 | |
| *** IlyaG has quit IRC | 21:59 | |
| JayF | clarkb: are the prereqs for the reproduce script docuemnted anywhere? | 21:59 |
| *** amotoki has joined #openstack-infra | 22:00 | |
| *** flwang has joined #openstack-infra | 22:00 | |
| clarkb | JayF: uh just what devstack-gate need | 22:00 |
| clarkb | it should mostly run | 22:00 |
| flwang | clarkb: sorry for bugging you again | 22:00 |
| JayF | ./reproduce.sh: line 69: virtualenv: command not found | 22:00 |
| *** asettle has quit IRC | 22:00 | |
| JayF | :) | 22:00 |
| flwang | clarkb: can you have work out the zaqar-ui-core issue for me? thanks | 22:00 |
| JayF | clarkb: that actually appears to be all it needed; virtualenv | 22:01 |
| anteaya | JayF: awesome, was going to give it a spin myself too | 22:01 |
| *** dims has quit IRC | 22:01 | |
| clarkb | flwang: you need group membership? | 22:01 |
| *** dims_ has joined #openstack-infra | 22:01 | |
| JayF | well it also needs a c compiler now | 22:01 |
| clarkb | flwang: what is the gerrit user to add? | 22:01 |
| clarkb | JayF: aha! we do need gcc for boostrapping (that was a question earlier today) | 22:01 |
| flwang | clarkb: flwang@catalyst.net.nz thanks | 22:01 |
| clarkb | greghaynes: ^ there you go | 22:01 |
| clarkb | JayF: we thought we might not actually need it, you have discovered otehrwise | 22:02 |
| JayF | clarkb: ./reproduce.sh: line 79: /usr/zuul-env/bin/zuul-cloner: No such file or directory | 22:02 |
| *** tphummel has quit IRC | 22:02 | |
| JayF | clarkb: that is one I'm going to have a bit hrder of a time finding | 22:02 |
| clarkb | JayF: it definitely attempts to make that one | 22:02 |
| JayF | hmm, so if it failed on c compliler | 22:02 |
| clarkb | JayF: you probably need to sudo the script | 22:02 |
| JayF | maybe remove the dir | 22:02 |
| clarkb | ah ya | 22:03 |
| clarkb | its does a -d | 22:03 |
| clarkb | so | 22:03 |
| JayF | a -e, in fact :P | 22:03 |
| clarkb | even better | 22:03 |
| clarkb | we should probably have the script self document these things in comments | 22:03 |
| JayF | that seems to have made it happier; so far only build-essential and virtualenv | 22:03 |
| JayF | and apparently even more stuff now | 22:03 |
| JayF | python-dev | 22:03 |
| clarkb | # run as root, have build-essential and python-dev installed and python-virtualenv | 22:03 |
| clarkb | JayF: ya thats required to pip install many things | 22:03 |
| anteaya | clarkb: that would help | 22:03 |
| JayF | oh god for a sec I thought you were saying that was already in the file | 22:04 |
| JayF | almost felt stupid there | 22:04 |
| clarkb | JayF: no I think it isn't but it should be! | 22:04 |
| *** fawadkhaliq has quit IRC | 22:04 | |
| anteaya | I agree | 22:04 |
| *** jamesmcarthur has joined #openstack-infra | 22:04 | |
| clarkb | but one thing at a time before I go crazy, zaqar-ui | 22:04 |
| *** tphummel has joined #openstack-infra | 22:04 | |
| JayF | ugh | 22:04 |
| JayF | needs git too | 22:04 |
| *** mylu has joined #openstack-infra | 22:04 | |
| JayF | clarkb: point me at the repo, I'll make this not-suck when I'm done | 22:04 |
| docaedo | anyone have a minute to log on to the app-catalog server and check to see if a file exists for me? | 22:04 |
| *** fawadkhaliq has joined #openstack-infra | 22:04 | |
| clarkb | JayF: its in openstack-infra/devstack-gate | 22:05 |
| JayF | git, python-dev, build-essential and pip install virtualenv so far :) | 22:05 |
| JayF | clarkb: cool, I'll put something up there if I get to the bottom | 22:05 |
| clarkb | JayF: I think that should be it | 22:05 |
| anteaya | JayF: I'll wait for you and then review your patch after you figure it out | 22:05 |
| clarkb | JayF: and dont forget to mention root | 22:05 |
| clarkb | flwang: ok done | 22:05 |
| flwang | clarkb: thank you so much :) | 22:05 |
| *** zul has joined #openstack-infra | 22:06 | |
| clarkb | krotscheck: did you get to the bottom of the firefox issues? | 22:08 |
| JayF | anteaya: clarkb: assuming I don't find more prereqs, something like this look sane? https://gist.github.com/jayofdoom/00e28c20dc45a55e784b | 22:08 |
| clarkb | JayF: yup | 22:09 |
| clarkb | we can worry about the fact that this also runs on centos and fedora later :) | 22:09 |
| JayF | Oh really, it does? | 22:09 |
| JayF | but like, in the gate, it's always trusty, right? | 22:09 |
| jroll | JayF: I wonder if "must have proper internets" is obvious | 22:09 |
| *** baoli has quit IRC | 22:10 | |
| anteaya | jroll: ha ha ha | 22:10 |
| clarkb | JayF: no it can be centos7 or fedora as well | 22:10 |
| clarkb | but most jobs are trusty | 22:10 |
| anteaya | JayF: thanks | 22:10 |
| JayF | clarkb: okay, cool, just making sure my assumption that the ipa -src job ran always in trusty was sane | 22:10 |
| jroll | anteaya: I'm somewhat serious, I've seen people rabbit hole behind a proxy etc | 22:10 |
| anteaya | oh yeah, that | 22:10 |
| anteaya | good point | 22:10 |
| anteaya | I'm so far removed from that myself | 22:11 |
| clarkb | JayF: for that it is. We don't pick them randomly | 22:11 |
| anteaya | I consider it fairytale | 22:11 |
| clarkb | just that devstack-gate does run on all 3 | 22:11 |
| JayF | cool, makes sense | 22:11 |
| jroll | maybe "assumes computer has proxy-less access to the internet on every port" would be a valid addition | 22:11 |
| *** jamesmcarthur has quit IRC | 22:11 | |
| clarkb | ubunt-trusty is being rebuilt with the new infra packages list element | 22:11 |
| openstackgerrit | Merged openstack/diskimage-builder: Reorder developer quickstart docs https://review.openstack.org/286997 | 22:11 |
| clarkb | jroll: it should honor http_proxy | 22:11 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/devstack-gate: Document prereqs/usage for reproduce script https://review.openstack.org/294308 | 22:11 |
| clarkb | but that might be getting too complicated for a short header comment | 22:11 |
| jroll | clarkb: not if it's unset | 22:11 |
| jroll | yeah | 22:11 |
| JayF | clarkb: assuming if I'm to "running devstack" there's no other prereqs ^ | 22:12 |
| clarkb | JayF: yup | 22:12 |
| jroll | idk, just thinking of other failure cases | 22:12 |
| clarkb | the problem with firewalls and proxies is every implementation is different | 22:12 |
| JayF | I mean, any documentation >> no documentation | 22:12 |
| clarkb | particularly when you start talking about not http | 22:12 |
| JayF | as long as it's correct, even if it's not comprehensive | 22:12 |
| JayF | if you feel strongly about it tell me on gerrit :P | 22:12 |
| *** akshai_ has quit IRC | 22:13 | |
| *** yamahata has joined #openstack-infra | 22:14 | |
| *** yamahata has quit IRC | 22:15 | |
| *** yamahata has joined #openstack-infra | 22:15 | |
| *** piet has quit IRC | 22:16 | |
| *** piet has joined #openstack-infra | 22:16 | |
| *** amotoki has quit IRC | 22:17 | |
| anteaya | JayF: rebuilding a node to test | 22:17 |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| *** zul has quit IRC | 22:18 | |
| *** amrith is now known as _amrith_ | 22:19 | |
| *** woodster_ has joined #openstack-infra | 22:20 | |
| *** asettle has joined #openstack-infra | 22:20 | |
| *** thorst has joined #openstack-infra | 22:21 | |
| *** smarcet has joined #openstack-infra | 22:21 | |
| smarcet | fungi: seems that there is some error at openstack.org redis instancec | 22:22 |
| smarcet | its throwing errors | 22:22 |
| smarcet | could u check? | 22:22 |
| fungi | smarcet: looking now | 22:22 |
| *** |-paul-| has quit IRC | 22:22 | |
| *** dims has joined #openstack-infra | 22:23 | |
| smarcet | ok i must take off but exception is clear | 22:23 |
| smarcet | exception | 22:23 |
| smarcet | 'Predis\Connection\ConnectionException' with message 'Error while | 22:23 |
| smarcet | reading line from the server [tcp://127.0.0.1:6378]' | 22:23 |
| smarcet | thx :) | 22:23 |
| *** baoli has joined #openstack-infra | 22:23 | |
| fungi | smarcet: i'm not seeing any errors in the /var/log/redis/redis-server.log but will keep looking | 22:23 |
| *** ihrachys has quit IRC | 22:24 | |
| fungi | it's definitely still listening on 6378/tcp | 22:24 |
| smarcet | could we restart it ? | 22:25 |
| *** ihrachys has joined #openstack-infra | 22:25 | |
| *** asettle has quit IRC | 22:25 | |
| *** thiagop has quit IRC | 22:25 | |
| fungi | is it possible the data being read from it was malformed? | 22:25 |
| *** dims_ has quit IRC | 22:26 | |
| fungi | the log is pretty much just status info about the number of clients connected, number of bytes in use | 22:26 |
| *** baoli has quit IRC | 22:26 | |
| clarkb | fungi: how much free memory do we have? | 22:26 |
| fungi | a few messages about closing idle connections | 22:26 |
| *** baoli has joined #openstack-infra | 22:27 | |
| clarkb | redis is a memory hog and can get crnaky when memory is low | 22:27 |
| fungi | using ~75% of 2gb ram, and a third of that is buffers/cache | 22:27 |
| clarkb | ok thats not bd | 22:27 |
| clarkb | *bad | 22:27 |
| *** mylu has quit IRC | 22:27 | |
| fungi | hasn't touched the 2gb of swap at all | 22:27 |
| smarcet | just started rite now | 22:27 |
| smarcet | weird | 22:27 |
| ianw | JayF: to really reproduce, i think you want to run -> http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n165 first | 22:27 |
| fungi | system load is negligible | 22:28 |
| fungi | no indication of any oom killer activity in dmesg | 22:28 |
| smarcet | well seems that is working again | 22:28 |
| smarcet | perhaps a peek | 22:28 |
| clarkb | gah I didn't restart the builder so it isn't using the new elements list | 22:28 |
| clarkb | when this build completes I will restart the builder and rebuild | 22:29 |
| maratoid | anteaya: smaller example - properties macro: http://paste.openstack.org/show/491015/ project: http://paste.openstack.org/show/491016/ , job template: http://paste.openstack.org/show/491017/ | 22:29 |
| maratoid | stack trace: http://paste.openstack.org/show/491020/ | 22:29 |
| clarkb | actually just going to restart it now so we don't use this build | 22:29 |
| fungi | smarcet: yeah, i'm checking cacti now to see if we spotted any spikes in activity | 22:29 |
| clarkb | nope nevermind there are a bunch of image builds | 22:29 |
| anteaya | maratoid: looking | 22:29 |
| clarkb | gah | 22:29 |
| fungi | smarcet: clarkb: memory use has picked up in the last 15 minutes or so: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2067&rra_id=all | 22:30 |
| maratoid | anteaya: sorry, that's 'parameter' macro. not property | 22:30 |
| *** bpokorny_ has joined #openstack-infra | 22:31 | |
| *** MarkAtwood has joined #openstack-infra | 22:31 | |
| fungi | smarcet: clarkb: that looks anomalous compared to the last weeks/months | 22:31 |
| smarcet | yeah i am not able to login in on www.openstack.org | 22:32 |
| smarcet | altought ref stack and translate works ok | 22:32 |
| clarkb | I manually killed the dib process to prevent it from making an image that isn't correct | 22:32 |
| smarcet | dev works | 22:32 |
| smarcet | so its something on the server | 22:32 |
| fungi | smarcet: clarkb: tcp connections have picked up too http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2066&rra_id=all | 22:32 |
| clarkb | fungi: possibly more use from translaton team? | 22:32 |
| clarkb | but I agree its picked up | 22:33 |
| fungi | not sure if those are piling up because they're hung waiting on something | 22:33 |
| smarcet | and redis errors still comming down from log | 22:33 |
| smarcet | how much ram has the node ? 2GB ? | 22:33 |
| fungi | i can try as redis restart to see if that makes a difference, but have a feeling it might be something else | 22:33 |
| smarcet | well thing is | 22:34 |
| fungi | yeah, 2gb but it's only got around 50% allocated and another 25% in use for buffers/cache at the moment | 22:34 |
| smarcet | if we restart all sessions will lost | 22:34 |
| smarcet | bc openstackid use redis to store sessions | 22:34 |
| smarcet | not big deal | 22:34 |
| *** bpokorny has quit IRC | 22:34 | |
| fungi | i'm going to take a look at the apache access logs to see if there might be a lot of connections from one or a handful of sources | 22:35 |
| *** camunoz has joined #openstack-infra | 22:35 | |
| anteaya | maratoid: so you must have a file that includes the contents -kraken-destroy or something | 22:35 |
| anteaya | maratoid: do you? | 22:35 |
| maratoid | yes | 22:35 |
| maratoid | each one of those is in a separate yaml files | 22:35 |
| smarcet | fungi got it | 22:36 |
| maratoid | job template, project and parameter macro | 22:36 |
| fungi | smarcet: a bunch of hits from 2001:4800:7820:103:be76:4eff:fe05:b574 in rackspace which has no reverse dns record | 22:36 |
| clarkb | smarcet: is the redis not configured to write to disk? if that happens a restart shouldn't kill any sessions | 22:36 |
| *** jamesmcarthur has quit IRC | 22:36 | |
| JayF | clarkb: Not only did that not reproduce the error, it doesn't even seem to have hit the error codepath at all | 22:38 |
| smarcet | clarkb: not sure had to review config settings on puppet | 22:39 |
| fungi | i have a feeling that ipv6 address is something to do with the www.openstack.org site | 22:39 |
| anteaya | maratoid: have you run other job templates successfully? | 22:39 |
| anteaya | maratoid: like this one: https://github.com/maratoid/kraken-ci-jobs/blob/master/kraken-run-k8s-density-tests/kraken-run-k8s-density-tests.yaml | 22:39 |
| fungi | also a bunch this hour from 2001:4800:7821:101:be76:4eff:fe06:858b which is also in rackspace and also has no reverse dns | 22:40 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:40 | |
| JayF | clarkb: anteaya: Yeah, I can't even see evidence it even tried to clone in the IPA repos | 22:40 |
| fungi | smarcet: clarkb whatever the spike in established tcp connections was, it's dying back off again | 22:41 |
| fungi | as is the memory spike | 22:41 |
| maratoid | anteaya: you know, I just tried updating to the latest version (i was at 1.4.0, installed through pip) and the minimal example actually worked. | 22:41 |
| maratoid | sorry, this whole thing is created by automation, and i think the part that's supposed to install from source is not quite working | 22:42 |
| clarkb | JayF: hrm that may mean the reproduce script is buggy | 22:42 |
| clarkb | JayF: like maybe it isn't writing out hook functions | 22:43 |
| *** matrohon has quit IRC | 22:43 | |
| *** jamesmcarthur has joined #openstack-infra | 22:43 | |
| JayF | idk, I'm going to use the vm to try and repro manually now | 22:43 |
| *** dingyichen has joined #openstack-infra | 22:43 | |
| JayF | it did install docker though, and it is working /o\ | 22:44 |
| JayF | so my vm isn't having the issue at all | 22:44 |
| JayF | I'm pretty sure it's some kind of race, so running it on a beefy server might make it disappear | 22:44 |
| JayF | but it didn't even run automatically so I don't even know that | 22:45 |
| smarcet | fungi: well not sure this start to happen 30 minutes or so ago | 22:46 |
| anteaya | JayF: :( | 22:46 |
| smarcet | the result is that www.openstack.org login is not working | 22:46 |
| anteaya | maratoid: yay | 22:46 |
| smarcet | fungi: not sure what to do | 22:46 |
| anteaya | maratoid: sounds like you are fixed for the templates bit? | 22:46 |
| *** mylu has joined #openstack-infra | 22:47 | |
| fungi | smarcet: some web searching on that error message indicates the application is running into a timeout reading from the redis socket | 22:47 |
| smarcet | yeah i saw it | 22:47 |
| fungi | smarcet: i'll go ahead and restart redis and we'll see if it recovers for very long | 22:47 |
| smarcet | they said increment time out | 22:47 |
| smarcet | ok | 22:48 |
| smarcet | yeah its the only alternative | 22:48 |
| maratoid | anteaya: i think so. seems like maybe they weren't actually broken :/ thanks! | 22:48 |
| smarcet | i am afraid | 22:48 |
| fungi | okay, it's restarted | 22:48 |
| fungi | i'm watching the application log for any more redis read errors | 22:49 |
| fungi | yep, there it is | 22:49 |
| fungi | still happening after the restart | 22:49 |
| pabelanger | ianw: still uploading images :( | 22:49 |
| fungi | a lot of "DelayCounterMeasure: attempt from banned ip" messages too | 22:49 |
| anteaya | maratoid: nice to meet you, happy trails | 22:49 |
| smarcet | still not working :/ | 22:49 |
| *** kaisers_ has joined #openstack-infra | 22:49 | |
| fungi | smarcet: 2001:4800:7820:103:be76:4eff:fe05:b574 is the one i saw hitting apache a bunch (in rackspace, no reverse dns) and it's what's being blocked by openstackid | 22:50 |
| ianw | pabelanger: alright, i'm uploading my test images too. | 22:50 |
| smarcet | yeah | 22:50 |
| smarcet | i know the issue | 22:50 |
| smarcet | was www.openstack.org | 22:50 |
| smarcet | got banned | 22:50 |
| smarcet | resolved | 22:50 |
| smarcet | horrray | 22:50 |
| smarcet | removed from banned ips | 22:50 |
| fungi | that's a fun failure mode | 22:51 |
| *** jamesmcarthur has quit IRC | 22:51 | |
| pabelanger | ianw: I don't know the whole story, but jeblair and fungi did some tests to at mid-cycle to increase the amount of image uploads I believe. I think 8 or so? | 22:51 |
| ianw | pabelanger: i still think we should have logs from the previous upload? | 22:52 |
| smarcet | fungi: yeah sorry should look on the banned table first | 22:52 |
| smarcet | sorry | 22:52 |
| smarcet | i will add www ip on the white list table | 22:52 |
| clarkb | pabelanger: ianw its still 4 upload threads, during the sprint we tested 16 and thats about what it took to saturate our link | 22:53 |
| smarcet | fungi: tnx for all your help today and sorry for false alarm | 22:53 |
| *** maratoid has quit IRC | 22:53 | |
| fungi | smarcet: glad it wasn't something more serious | 22:53 |
| *** smarcet has quit IRC | 22:53 | |
| pabelanger | clarkb: is there anything stopping us from getting a patch up today to bump it close to 16? | 22:54 |
| *** hashar has quit IRC | 22:54 | |
| fungi | smarcet: also, getting reverse dns entries added for those systems would be a huge help. we set them on all of our servers, so i was for example able to see that the #3 source of traffic to openstackid.org was translate.openstack.org, but #1 and #2 were just "random hosts in rackspace) | 22:54 |
| *** tqtran is now known as tqtran-afk | 22:54 | |
| clarkb | greghaynes: looks like ubuntu-minimal inclues gcc too fwiw | 22:54 |
| *** kaisers_ has quit IRC | 22:54 | |
| fungi | oh well, he's gone | 22:54 |
| *** ihrachys has quit IRC | 22:54 | |
| clarkb | pabelanger: I don't think so, just someone needs to write the chang eand get it merged | 22:54 |
| clarkb | we also need to make the config reload on changes | 22:54 |
| pabelanger | right | 22:54 |
| *** mriedem has joined #openstack-infra | 22:54 | |
| *** Sukhdev has joined #openstack-infra | 22:54 | |
| pabelanger | okay, I can see what is needed | 22:54 |
| *** Keedya has joined #openstack-infra | 22:55 | |
| *** ihrachys has joined #openstack-infra | 22:55 | |
| greghaynes | clarkb: wah | 22:55 |
| greghaynes | clarkb: I wonder what pulls it in | 22:55 |
| *** dimtruck is now known as zz_dimtruck | 22:56 | |
| *** Keedya has quit IRC | 22:57 | |
| greghaynes | clarkb: it shouldnt be debootstrap, we use minbase there... Also, build-essential is more than gcc | 22:58 |
| fungi | if smarcet comes back, 2001:4800:7820:103:be76:4eff:fe05:b574 is still being blocked by openstackid.org | 22:58 |
| fungi | or started getting blocked again, anyway | 22:58 |
| *** rbrndt has quit IRC | 22:58 | |
| *** baoli has quit IRC | 22:59 | |
| clarkb | greghaynes: it looks an awful lot like debootstrap | 22:59 |
| clarkb | its one of the first things that happens | 22:59 |
| clarkb | it hasn't gotten to my new package install stuff yet | 22:59 |
| *** baoli has joined #openstack-infra | 23:00 | |
| *** kaisers_ has joined #openstack-infra | 23:00 | |
| *** piet has quit IRC | 23:00 | |
| *** HeOS has quit IRC | 23:00 | |
| *** piet has joined #openstack-infra | 23:01 | |
| *** sridhar_ram1 has joined #openstack-infra | 23:01 | |
| greghaynes | clarkb: hrm, maybe minbase does include gcc but not build-essential | 23:01 |
| *** tqtran-afk has quit IRC | 23:01 | |
| greghaynes | clarkb: which is funny because I think cloud images do not include gcc | 23:01 |
| fungi | that sounds right | 23:01 |
| *** sridhar_ram has quit IRC | 23:01 | |
| fungi | gcc is considered essential on debian iirc | 23:01 |
| *** jamesmcarthur has joined #openstack-infra | 23:01 | |
| greghaynes | ah | 23:01 |
| Clint | fungi: no, build-essential | 23:02 |
| fungi | build-essential is a metapackage of things you can install if you intend to compile typical c software, but having gcc is useful for more than just general build work | 23:02 |
| fungi | Clint: it's just in build-essential? | 23:02 |
| Clint | yup | 23:02 |
| fungi | double checking | 23:02 |
| fungi | then does debootstrap special-case gcc? | 23:02 |
| *** HeOS has joined #openstack-infra | 23:03 | |
| fungi | right you are, Priority: optional | 23:03 |
| *** cody-somerville has quit IRC | 23:03 | |
| greghaynes | oh hey, minbase does include build-essential, I was way off | 23:03 |
| *** cody-somerville has joined #openstack-infra | 23:04 | |
| fungi | okay, so we're already ending up with all of build-essential on the minimal images? | 23:04 |
| clarkb | http://paste.openstack.org/show/491024/ there it goes | 23:04 |
| greghaynes | maybe someone else can help parse this debootstrap manpage | 23:05 |
| clarkb | I think this is going to work fine, I will do my best to restart nodepool-builder when the current uploads finish | 23:05 |
| *** baoli has quit IRC | 23:05 | |
| greghaynes | I think it is saying that build-essential is in minbase, but everytime I parse it I get more confused | 23:05 |
| *** cfarquhar has joined #openstack-infra | 23:05 | |
| *** cfarquhar has quit IRC | 23:05 | |
| *** cfarquhar has joined #openstack-infra | 23:05 | |
| *** baoli has joined #openstack-infra | 23:05 | |
| *** yamahata has quit IRC | 23:06 | |
| greghaynes | ok, I think it is saying buildd includes build-essential, minbase does not | 23:06 |
| clarkb | fungi: well we always explicitly added build-essential (we need it for zuul cloner install) | 23:06 |
| *** kaisers_ has quit IRC | 23:06 | |
| clarkb | fungi: but this shows we odn't need to because ubuntu-minimal adds it in anways | 23:06 |
| fungi | greghaynes: mine says you need --variant=buildd to get build-essential | 23:06 |
| greghaynes | fungi: gotcha | 23:06 |
| greghaynes | so ya, youall must be adding that somehow | 23:07 |
| clarkb | we don't do anything other than use ubuntu-minimal | 23:07 |
| fungi | and that --variant=minbase "only includes essential packages and apt" | 23:07 |
| fungi | oh, though i'm looking on debian | 23:07 |
| * greghaynes kicks off a build to grep logs on | 23:07 | |
| fungi | maybe ubuntu makes their own rules for debootstrap variants | 23:07 |
| *** asettle has joined #openstack-infra | 23:07 | |
| *** baoli has quit IRC | 23:08 | |
| *** dims has quit IRC | 23:08 | |
| *** baoli has joined #openstack-infra | 23:08 | |
| greghaynes | Extracting gcc-4.9-base... | 23:10 |
| greghaynes | from the debootstrap output | 23:10 |
| clarkb | 22 images to upload | 23:11 |
| clarkb | so slow | 23:11 |
| *** jamesmcarthur has quit IRC | 23:11 | |
| *** moravec has joined #openstack-infra | 23:12 | |
| *** ihrachys has quit IRC | 23:13 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add more clients when checking for openstackclient command overlap https://review.openstack.org/294325 | 23:13 |
| *** ihrachys has joined #openstack-infra | 23:14 | |
| *** piet has quit IRC | 23:14 | |
| *** pvaneck has quit IRC | 23:15 | |
| jamielennox | AJaeger, ianw: i simplified https://review.openstack.org/#/c/271128/ would you be able to give it a look so i can move on to the dependant patches? | 23:15 |
| *** IlyaG has joined #openstack-infra | 23:15 | |
| clarkb | greghaynes: I don't think its a huge deal but wanted to point out its not as simple as us not installing it | 23:15 |
| cody-somerville | Latest gcc-x.y-base is priority required due to the dependency chain libc6 → libgcc1 → gcc-x.y-base | 23:16 |
| greghaynes | clarkb: Ya, I mostly am trying to make sure that -minimal doesnt ever have packages which don't exist in the cloud images | 23:16 |
| *** IlyaG has quit IRC | 23:17 | |
| *** lucasagomes has quit IRC | 23:18 | |
| *** geekinutah has joined #openstack-infra | 23:20 | |
| *** geekinutah has quit IRC | 23:21 | |
| *** jamesmcarthur has joined #openstack-infra | 23:22 | |
| *** IlyaG has joined #openstack-infra | 23:22 | |
| *** mylu has quit IRC | 23:22 | |
| *** adrian_otto has joined #openstack-infra | 23:22 | |
| ianw | jamielennox: ok ... 12 revisions make me think it's something more complicated but it's really just for adding quick nv tests? | 23:23 |
| jamielennox | ianw: there have been so many merge conflicts | 23:23 |
| jamielennox | ianw: the goal is to run non voting jobs with keystone v2 disabled | 23:23 |
| jamielennox | against other projects | 23:23 |
| *** sdake has joined #openstack-infra | 23:24 | |
| ianw | cool, just makes it look more complex than it is :) | 23:24 |
| *** mylu has joined #openstack-infra | 23:24 | |
| *** ihrachys has quit IRC | 23:24 | |
| *** Sukhdev has quit IRC | 23:24 | |
| jamielennox | ianw: this review also used to contain installing it into the other services, which was going to require +1s from everyone | 23:25 |
| jamielennox | now it's split up so we can hopefuly move it forward | 23:25 |
| *** lucasagomes has joined #openstack-infra | 23:25 | |
| *** Qiming has joined #openstack-infra | 23:25 | |
| *** ihrachys has joined #openstack-infra | 23:26 | |
| *** dims has joined #openstack-infra | 23:28 | |
| *** IlyaG has quit IRC | 23:30 | |
| *** IlyaG has joined #openstack-infra | 23:30 | |
| *** fawadkhaliq has quit IRC | 23:31 | |
| *** bpokorny_ has quit IRC | 23:31 | |
| *** fawadk has joined #openstack-infra | 23:31 | |
| *** bpokorny has joined #openstack-infra | 23:32 | |
| *** bpokorny has quit IRC | 23:32 | |
| *** dims has quit IRC | 23:32 | |
| *** bpokorny has joined #openstack-infra | 23:33 | |
| *** tphummel has quit IRC | 23:33 | |
| *** Sukhdev has joined #openstack-infra | 23:34 | |
| *** tphummel has joined #openstack-infra | 23:34 | |
| *** IlyaG has quit IRC | 23:35 | |
| *** Sukhdev has quit IRC | 23:36 | |
| *** jamesmcarthur has quit IRC | 23:36 | |
| fungi | since our job volume has died down, i'm going to temporarily delete the accumulated devstack-trusty nodes. hopefully once the post pipeline backlog catches up we can find a quiet moment to restart zuul and clear the gearman registrations | 23:36 |
| clarkb | mnaser: looks like vexxhost image uploads may be doing that broken caching thing again | 23:36 |
| *** baoli has quit IRC | 23:37 | |
| mnaser | clarkb: openstack-infra nagios red :p | 23:38 |
| mnaser | checking | 23:38 |
| pabelanger | clarkb: Ya, been like that for a few hours now | 23:38 |
| *** ihrachys has quit IRC | 23:39 | |
| *** SumitNaiksatam has quit IRC | 23:39 | |
| pabelanger | ianw: centos7 uploaded to rax-ord now | 23:39 |
| *** jamesmcarthur has joined #openstack-infra | 23:40 | |
| anteaya | fungi: yay | 23:40 |
| *** ihrachys has joined #openstack-infra | 23:40 | |
| *** alex_xu has quit IRC | 23:41 | |
| fungi | i missed the broken caching thing | 23:43 |
| *** eliqiao has quit IRC | 23:43 | |
| clarkb | fungi: where nginx caches in front of glance and clients get confused when all the bits are acceptd but no 200 is received for another 15 minutes | 23:44 |
| fungi | oh, right, that | 23:44 |
| fungi | timeout between the proxy and the backend | 23:44 |
| *** baoli has joined #openstack-infra | 23:44 | |
| *** moravec has quit IRC | 23:45 | |
| fungi | after it's all buffered | 23:45 |
| *** eliqiao has joined #openstack-infra | 23:45 | |
| ianw | pabelanger: ok, is it registered? | 23:46 |
| *** tpsilva has quit IRC | 23:47 | |
| pabelanger | ianw: maybe? | 23:47 |
| pabelanger | I see 162.242.209.98 with deleted state | 23:47 |
| pabelanger | and another building | 23:47 |
| pabelanger | let me monitor | 23:47 |
| pabelanger | so, it got an IP | 23:48 |
| dmsimard | nginx buffering uploads, I've had to deal with that before .. had to settle with an nginx fork (tengine) by the alibaba guys, they have a feature to disable body buffering | 23:48 |
| *** fverbs has joined #openstack-infra | 23:48 | |
| pabelanger | ianw: looks like SSH timeout | 23:48 |
| clarkb | dmsimard: nginx can do it too under certain circumstainces | 23:48 |
| pabelanger | 2016-03-17 23:45:05,838 DEBUG nodepool.NodeLauncher: Node id: 8847509 is running, ipv4: 162.242.209.98, ipv6: 2001:4802:7800:1:be76:4eff:fe21:3d0f | 23:49 |
| pabelanger | 2016-03-17 23:45:05,838 DEBUG nodepool.NodeLauncher: Node id: 8847509 testing ssh at ip: 162.242.209.98 | 23:49 |
| dmsimard | interesting, things might have changed since I last looked at this | 23:49 |
| pabelanger | ianw: ^ | 23:49 |
| *** baoli has quit IRC | 23:49 | |
| ianw | pabelanger: alright, that's pretty much matching what i expected from my testing yesterday | 23:49 |
| pabelanger | also, sorry for pastebomb | 23:49 |
| ianw | my upload with glean changes is still churning it's way through | 23:49 |
| ianw | in short, i think we're taking about 2 minutes to timeout on dhcp before we bring the interface up | 23:50 |
| *** alex_xu has joined #openstack-infra | 23:50 | |
| *** baoli has joined #openstack-infra | 23:52 | |
| jroll | infra-root, could someone hold ubuntu-trusty-ovh-bhs1-8847643 for me please? | 23:52 |
| clarkb | jroll: done | 23:52 |
| jroll | ssh key: http://paste.openstack.org/show/491028/ | 23:52 |
| jroll | thank you clarkb | 23:53 |
| *** davideagnello has quit IRC | 23:53 | |
| *** fedexo has joined #openstack-infra | 23:53 | |
| clarkb | jroll: ssh jenkins@158.69.90.73 | 23:53 |
| jroll | ty sir | 23:53 |
| *** davideagnello has joined #openstack-infra | 23:54 | |
| *** mylu has quit IRC | 23:54 | |
| *** markvoelker has quit IRC | 23:54 | |
| *** baoli has quit IRC | 23:54 | |
| *** angdraug has quit IRC | 23:55 | |
| *** baoli has joined #openstack-infra | 23:55 | |
| *** jamesmcarthur has quit IRC | 23:57 | |
| *** MarkAtwood has quit IRC | 23:58 | |
| *** dalgaaf has quit IRC | 23:58 | |
| *** davideagnello has quit IRC | 23:59 | |
| *** ajmiller has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!