Monday, 2014-03-24

*** saschpe- has quit IRC00:00
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Unbound: also override dhclient config on debian
jeblairlifeless: i think that will fix it ^00:02
jeblairclarkb, mordred, fungi: ^00:02
*** rhsu has joined #openstack-infra00:02
jeblairlifeless: i think that even without the unbound change, that server would be using the dhcp-supplied nameserver00:05
lifelessjeblair: it will have two nameservers00:06
lifeless192.168.1.1 and
lifeless1.1 will be a brain damaged dnsmasq00:06
lifelesse.g. nameserver
lifelesswould be the default without your scripts00:06
*** saschpe has joined #openstack-infra00:07
jeblairlifeless: i'm guessing the change is that we have disabled resolvconf, and perhaps dhclient is overwriting the nameserver when it gets the second lease whereas resolvconf may have added it.00:09
jeblairsince comes from the eth0 lease and comes from the eth1 lease00:09
*** yamahata__ has quit IRC00:09
*** dcramer_ has quit IRC00:09
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Unbound: also override dhclient config on debian
lifelessjeblair: disabling resolvconf - that worries me, because its exactly this sort of multi-interface thing where resolvconf solves lots and lots of problems00:11
*** thedodd has joined #openstack-infra00:12
jeblairlifeless: unfortunately, rackspace performs (or has performed) file injection on /etc/resolv.conf which breaks the symlink00:13
*** CaptTofu has joined #openstack-infra00:13
jeblairlifeless: if the symlink is broken, it's disabled00:13
*** caleb_ has joined #openstack-infra00:13
jeblairlifeless: so we ended up with a bunch of machines where it was not easy to make them consistent00:13
nibalizermordred: whats the reason to have the slaves run puppet apply instead of puppet agent?00:13
jeblairlifeless: at any rate, we only want one thing in our resolv.conf files "nameserver", so it seemed the better way to go is to make that a static file, and disable resolvconf in all cases00:14
*** krotscheck has quit IRC00:14
jeblairlifeless: (though it does also apparently mean that we need to prevent dhclient from writing the file)00:14
openstackgerritA change was merged to openstack-infra/config: Unbound: also override dhclient config on debian
*** krotscheck has joined #openstack-infra00:15
jeblairlifeless: any chance you can fix the quota?  i can't build an image when we're over quota00:16
lifelessjeblair: I'm hoping to hear from mikal about what data he needs to examine the problem00:17
jeblairlifeless: okay, retrying managed to find a slot00:18
lifelessjeblair: btw you don't need to remove resolvconf to have a static /etc/resolv.conf00:18
jeblairlifeless: we didn't, we just broke the symlink00:19
lifelessjeblair: since resolvconf's database is separate; just removing the symlink is all you need.00:19
lifelessjeblair: ok00:19
lifelessjeblair: I haven't read your changes obviously ;)00:19
lifelessI'm surprised dhclient writes the file, since with resolvconf present it should never do that, rather push the data to resolvconf and let resolvconf DTRT00:20
jeblairlifeless: me too00:20
jeblairclarkb, mordred, fungi: ^ we should keep an eye on what happens after the tripleo image build is finished; if it does in fact fix the problem or make it worse00:20
jeblairclarkb, mordred, fungi: because it's also likely to have an effect on hpcloud precise and devstack-precise nodes00:21
*** harlowja_at_home has quit IRC00:21
lifelessjeblair: are you running your own caching nameserver on each node? Wouldn't it be better to run a caching nameserver in each cloud ?00:21
lifelessjeblair: we were going to put a recursive bind9 in ci-overcloud00:21
lifeless(and point DNS at that)00:22
*** bhuvan has joined #openstack-infra00:22
jeblairlifeless: each node is running a caching-forwarding nameserver; having a dedicated per-region caching-recursive nameserver is probably an option when all our providers support neutron00:22
lifelessjeblair: where does it forward to? That dhclient override squashes the DHCP supplies NS, right ?00:23
jeblairlifeless: but until then, running a public recursive nameserver is a bit too much of a commitment (esp with dns amplication attacks, etc, these days)00:23
jeblairlifeless: the supplied nameservers if they are well-behaved (eg rax), or google if they are not (hp)00:23
lifelessjeblair: oh, I wasn't going to run a public one :)00:23
lifelessjeblair: I was joing to put it on a node inside the cloud00:24
jeblairlifeless: i'm not either, but without neutron support, we can't run a network-private one (because we don't have a private network)00:24
*** matsuhashi has joined #openstack-infra00:25
lifelessjeblair: do you still get the dns dynamically, or do you manually store it in the puppet rules?00:25
lifelessjeblair: I'm asking so if/when we get these local cached resources in ci-overcloud we know whether to change our DHCP, or to just tell you.00:26
*** bhuvan has quit IRC00:26
mordrednibalizer: the slaves are single use slaves00:27
mordrednibalizer: we create and destroy as many as 20k of them in a day00:27
mordrednibalizer: which means that the cert stuff on the puppemaster would be a complete nightmare00:28
mordrednibalizer: also, we consider them hostile00:28
mordrednibalizer: make sense?00:29
mordredjeblair: kk00:29
mordrednibalizer: needless to say - puppet apply is SUPER important and breaking it is a non-starter :)00:30
*** rhsu has quit IRC00:32
jeblairlifeless: dynamically00:40
lifelessjeblair: ok, cool00:41
*** jeremyb has quit IRC00:43
*** jeremyb has joined #openstack-infra00:43
*** Ryan_Lane has quit IRC00:50
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add expand by default option to status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add a filter box to status.html
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs
*** jhesketh has quit IRC00:54
*** Ryan_Lane has joined #openstack-infra00:55
*** jhesketh has joined #openstack-infra00:55
*** yongli has joined #openstack-infra01:02
*** BadCub01 has quit IRC01:12
*** yamahata has joined #openstack-infra01:14
jeblairclarkb, mordred, fungi: dns on new tripleo nodes look good, so i think worked as intended.  i expect hpcloud to be fine, but just in case, be aware of that in case we need to kill new images tomorrow morning.01:20
jeblairlifeless: ^01:20
*** thedodd has quit IRC01:20
lifelessjeblair: cool, thank you!01:22
*** CaptTofu has quit IRC01:23
nibalizermordred: okay, that makes sense01:26
*** mburned has quit IRC01:32
*** nosnos has joined #openstack-infra01:36
*** thuc has joined #openstack-infra01:41
*** thuc_ has joined #openstack-infra01:43
*** thuc has quit IRC01:47
*** mburned has joined #openstack-infra01:49
*** mburned has quit IRC01:56
*** mriedem has quit IRC01:56
*** mburned has joined #openstack-infra02:00
*** mriedem has joined #openstack-infra02:07
*** yjiang has joined #openstack-infra02:07
*** thedodd has joined #openstack-infra02:08
*** adalbas has joined #openstack-infra02:12
*** david-lyle has joined #openstack-infra02:14
*** CaptTofu has joined #openstack-infra02:15
*** mburned has quit IRC02:17
*** mburned has joined #openstack-infra02:18
*** zehicle_at_dell has quit IRC02:21
*** zehicle_at_dell has joined #openstack-infra02:22
*** mburned has quit IRC02:23
*** mriedem has quit IRC02:30
*** caleb_ has quit IRC02:32
*** mburned_out has joined #openstack-infra02:34
*** mburned_out has quit IRC02:42
*** thuc_ has quit IRC02:43
*** thuc has joined #openstack-infra02:43
*** thuc_ has joined #openstack-infra02:45
*** bknudson has quit IRC02:45
*** thuc has quit IRC02:47
openstackgerritA change was merged to openstack-infra/pypi-mirror: Fix run-mirror writing wrong project directory
*** thedodd has quit IRC02:48
*** rcleere has joined #openstack-infra02:52
*** matsuhashi has quit IRC02:55
*** adalbas has quit IRC03:03
*** michchap_ has joined #openstack-infra03:08
*** michchap has quit IRC03:09
*** CaptTofu has quit IRC03:11
*** CaptTofu has joined #openstack-infra03:12
*** CaptTofu has quit IRC03:17
*** harlowja_at_home has joined #openstack-infra03:20
*** matsuhashi has joined #openstack-infra03:21
*** nati_ueno has joined #openstack-infra03:23
*** Ryan_Lane has quit IRC03:31
*** harlowja_at_home has quit IRC03:41
*** ekarlso has quit IRC03:48
*** ekarlso has joined #openstack-infra03:48
*** thuc_ has quit IRC03:53
*** thuc has joined #openstack-infra03:54
*** thuc has quit IRC03:59
*** thuc has joined #openstack-infra04:02
*** todd_dsm has joined #openstack-infra04:05
*** bhuvan has joined #openstack-infra04:06
*** bhuvan has quit IRC04:16
*** harlowja_at_home has joined #openstack-infra04:20
*** _Alexandra_ has joined #openstack-infra04:27
*** Alexandra has quit IRC04:30
*** _Alexandra_ has quit IRC04:30
openstackgerritKhai Do proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o
openstackgerritKhai Do proposed a change to openstack-infra/config: update gerrit project configs to use new Workflow label
openstackgerritKhai Do proposed a change to openstack-infra/config: update trigger to match new workflow label
*** thuc_ has joined #openstack-infra04:39
*** Alexandra has joined #openstack-infra04:40
*** todd_dsm has quit IRC04:41
*** thuc has quit IRC04:42
*** alff has joined #openstack-infra04:42
*** thuc_ has quit IRC04:43
*** jepoy_ has joined #openstack-infra04:44
*** jepoy has quit IRC04:45
*** mestery has joined #openstack-infra04:46
*** nati_ueno has quit IRC04:46
*** nati_ueno has joined #openstack-infra04:47
*** nati_ueno has quit IRC04:51
*** nati_ueno has joined #openstack-infra04:52
*** harlowja_at_home has quit IRC05:06
*** bhuvan has joined #openstack-infra05:12
*** CaptTofu has joined #openstack-infra05:12
*** chandankumar has quit IRC05:14
*** mestery has quit IRC05:14
*** nosnos_ has joined #openstack-infra05:16
*** CaptTofu has quit IRC05:17
*** bhuvan has quit IRC05:19
*** nosnos has quit IRC05:19
*** nicedice has quit IRC05:23
*** zhiyan_ is now known as zhiyan05:24
*** chandan_kumar has joined #openstack-infra05:25
openstackgerritZhiQiang Fan proposed a change to openstack/requirements: Rename PrettyTable to prettytable
*** sdake_ has quit IRC05:47
yjianghi guys,I want to build a openstack 3rd part testing system.Should I use zuul or gerrit trigger as event stream trigger?Is there any related document about these details of configuration?05:48
*** alff has quit IRC05:50
*** todd_dsm has joined #openstack-infra05:52
*** mihgen has joined #openstack-infra06:00
*** matsuhas_ has joined #openstack-infra06:00
*** matsuhashi has quit IRC06:01
*** Alexandra has quit IRC06:01
*** mestery has joined #openstack-infra06:13
*** mestery has quit IRC06:13
*** mestery has joined #openstack-infra06:14
*** nati_ueno has quit IRC06:14
*** mestery_ has joined #openstack-infra06:15
*** mestery has quit IRC06:15
*** nosnos has joined #openstack-infra06:19
*** mestery_ has quit IRC06:20
*** nosnos_ has quit IRC06:22
*** obondarev has quit IRC06:25
*** ildikov_ has quit IRC06:26
*** matsuhas_ has quit IRC06:31
*** matsuhashi has joined #openstack-infra06:31
*** saju_m has joined #openstack-infra06:35
openstackgerritThomas Goirand proposed a change to openstack/requirements: Using happybase 0.8
*** moted has quit IRC06:38
*** amotoki has joined #openstack-infra06:41
*** mrda is now known as mrda_away06:45
*** jamielennox is now known as jamielennox|away06:51
*** saju_m has quit IRC06:59
*** saju_m has joined #openstack-infra07:12
*** CaptTofu has joined #openstack-infra07:13
openstackgerritIvan Melnikov proposed a change to openstack-dev/hacking: Disable logging correctly
*** ildikov_ has joined #openstack-infra07:13
*** saju_m has quit IRC07:17
*** saju_m has joined #openstack-infra07:17
*** michchap has joined #openstack-infra07:18
*** branen_ has joined #openstack-infra07:18
*** mihgen_ has joined #openstack-infra07:18
*** CaptTofu has quit IRC07:18
*** kmartin has joined #openstack-infra07:18
*** krotscheck has quit IRC07:19
*** nprivalova has quit IRC07:19
*** todd_dsm has quit IRC07:19
*** jeremyb has quit IRC07:19
*** branen__ has quit IRC07:19
*** KurtMartin has quit IRC07:19
*** jhesketh_ has quit IRC07:19
*** loquacities has quit IRC07:19
*** Steap has quit IRC07:19
*** michchap_ has quit IRC07:19
*** Steap_ has joined #openstack-infra07:19
*** jeremyb_ has joined #openstack-infra07:19
*** jhesketh_ has joined #openstack-infra07:19
*** mihgen has quit IRC07:20
*** mihgen_ is now known as mihgen07:20
*** loquacities has joined #openstack-infra07:20
*** saju_m has quit IRC07:22
*** jeremyb_ has quit IRC07:23
*** jeremyb_ has joined #openstack-infra07:23
*** saju_m has joined #openstack-infra07:23
*** jeremyb_ is now known as jeremyb07:24
*** krotscheck has joined #openstack-infra07:24
*** saju_m has quit IRC07:30
*** saju_m has joined #openstack-infra07:31
*** nprivalova has joined #openstack-infra07:33
*** flaper87|afk is now known as flaper8707:40
*** jlibosva has joined #openstack-infra07:44
*** e0ne has joined #openstack-infra07:44
*** e0ne has quit IRC07:47
*** ttx has quit IRC07:51
*** todd_dsm has joined #openstack-infra07:57
*** bookwar has joined #openstack-infra08:02
*** ttx has joined #openstack-infra08:02
*** ttx has quit IRC08:02
*** ttx has joined #openstack-infra08:02
*** ttx has quit IRC08:06
*** ttx has joined #openstack-infra08:07
*** ttx has quit IRC08:07
*** ttx has joined #openstack-infra08:07
*** dizquierdo has joined #openstack-infra08:08
*** yolanda has joined #openstack-infra08:08
*** rlandy has joined #openstack-infra08:11
*** jpich has joined #openstack-infra08:21
*** bogdando has joined #openstack-infra08:26
*** hashar has joined #openstack-infra08:28
*** e0ne has joined #openstack-infra08:30
*** yolanda has quit IRC08:31
*** afazekas has quit IRC08:32
*** afazekas has joined #openstack-infra08:33
*** jgallard has joined #openstack-infra08:34
*** soren_ has quit IRC08:36
*** soren_ has joined #openstack-infra08:36
*** soren_ is now known as soren08:36
*** yolanda has joined #openstack-infra08:36
*** thuc has joined #openstack-infra08:39
*** roeyc has joined #openstack-infra08:41
*** pblaho has joined #openstack-infra08:43
*** thuc has quit IRC08:44
*** hashar has quit IRC08:46
*** mihgen has quit IRC08:50
*** ilyashakhat has quit IRC08:55
*** derekh has joined #openstack-infra08:57
*** jlibosva has quit IRC08:58
*** jepoy_ has quit IRC08:58
*** hashar has joined #openstack-infra08:59
*** rossella_s has joined #openstack-infra08:59
*** jlibosva has joined #openstack-infra09:00
*** yassine has joined #openstack-infra09:05
*** ihrachys|wfh is now known as ihrachys09:06
*** jooools has joined #openstack-infra09:09
*** jepoy has joined #openstack-infra09:13
*** jlibosva has quit IRC09:13
*** jlibosva has joined #openstack-infra09:14
*** CaptTofu has joined #openstack-infra09:14
openstackgerritVictor Stinner proposed a change to openstack/requirements: Allow trollius 0.2
*** CaptTofu has quit IRC09:19
*** ilyashakhat has joined #openstack-infra09:20
*** dpyzhov has joined #openstack-infra09:24
*** pbelanyi has joined #openstack-infra09:24
*** katyafervent has joined #openstack-infra09:25
*** jepoy has quit IRC09:26
*** e0ne_ has joined #openstack-infra09:27
*** johnthetubaguy has joined #openstack-infra09:27
*** johnthetubaguy has quit IRC09:28
*** johnthetubaguy has joined #openstack-infra09:28
*** mihgen has joined #openstack-infra09:29
*** e0ne has quit IRC09:29
*** dizquierdo has quit IRC09:31
*** ominakov has joined #openstack-infra09:33
*** vkozhukalov has joined #openstack-infra09:50
*** ociuhandu has quit IRC09:53
*** johnthetubaguy1 has joined #openstack-infra09:55
*** johnthetubaguy has quit IRC09:56
*** yjiang has quit IRC09:56
*** ociuhandu has joined #openstack-infra10:08
*** vkozhukalov has left #openstack-infra10:11
*** mkoderer has joined #openstack-infra10:23
*** jp_at_hp has joined #openstack-infra10:28
*** matsuhashi has quit IRC10:33
*** matsuhas_ has joined #openstack-infra10:35
*** david-lyle has quit IRC10:41
*** yamahata has quit IRC10:43
*** CaptTofu has joined #openstack-infra10:48
*** yassine has quit IRC10:52
*** vkozhukalov has joined #openstack-infra10:54
*** dizquierdo has joined #openstack-infra11:00
*** mburned_out has joined #openstack-infra11:05
*** matsuhas_ has quit IRC11:07
ihrachysdoes anyone know whether any service is running on gate machines listening on :9000? We have a test failing due to this in stable/havana oslo-incubator.11:12
*** jepoy has joined #openstack-infra11:12
*** mbacchi has joined #openstack-infra11:14
openstackgerritRomain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions
*** jasondotstar has quit IRC11:20
*** nosnos has quit IRC11:23
*** jhesketh has quit IRC11:27
*** chandankumar_ has joined #openstack-infra11:29
*** dkranz has quit IRC11:30
*** sergmelikyan has joined #openstack-infra11:32
*** gnuoy has joined #openstack-infra11:32
sergmelikyanGuys, gate job is hang on gate-tempest-dsvm-neutron11:32
sergmelikyan*hang = stuck11:33
sergmelikyanAny help?11:33
*** salv-orlando has quit IRC11:37
*** _ruhe_ has joined #openstack-infra11:37
*** _ruhe_ has quit IRC11:38
*** _ruhe_ has joined #openstack-infra11:38
*** gmoro has quit IRC11:42
*** mburned_out is now known as mburned11:43
*** sandywalsh has joined #openstack-infra11:46
openstackgerritA change was merged to openstack/requirements: Updated oslo.messaging requirement
*** weshay has joined #openstack-infra11:46
*** sandywalsh has quit IRC11:50
*** jgallard has quit IRC11:53
*** matsuhashi has joined #openstack-infra11:53
*** nosnos has joined #openstack-infra11:54
*** gmoro has joined #openstack-infra11:54
ihrachyslifeless: ping re port 9000 on gate machines11:55
*** ianw- has joined #openstack-infra12:01
*** ianw has quit IRC12:01
*** jamielenz has joined #openstack-infra12:01
*** ianw- is now known as ianw12:01
*** sandywalsh has joined #openstack-infra12:03
*** jamielennox|away has quit IRC12:03
*** rfolco has joined #openstack-infra12:05
*** yassine has joined #openstack-infra12:05
*** pdmars has joined #openstack-infra12:06
*** yolanda has quit IRC12:07
sdaguesergmelikyan: why did you think it was hung?12:09
sergmelikyanNo new messages in console output for about 30m12:09
sdagueit merged at 7:4612:10
*** jepoy has quit IRC12:10
sdague11:46 UTC12:10
*** pdmars has quit IRC12:10
*** salv-orlando has joined #openstack-infra12:11
*** chandankumar_ has quit IRC12:13
*** pdmars has joined #openstack-infra12:13
*** matsuhashi has quit IRC12:14
*** yassine has quit IRC12:15
*** matsuhas_ has joined #openstack-infra12:17
*** yassine has joined #openstack-infra12:17
ihrachyssdague: do you have insight into software running on gate machines?12:17
*** aysyd has joined #openstack-infra12:19
*** ArxCruz has joined #openstack-infra12:21
sdagueihrachys: for what kinds of tests?12:21
ihrachyssdague: wsgi test that starts a service listening on :900012:22
*** dkranz has joined #openstack-infra12:22
ihrachyssdague: sometimes it suceeded, sometimes fail12:22
cyeohsdague: hi - re:  - is that -2 a "don't want this sort of functionality" or just want it to interact it ok with xen overriding it?12:23
sdagueusing a common port like that seems really error prone. If the test really needs to bind to a port, it needs to look for a free port as part of the test12:24
*** e0ne_ has quit IRC12:24
sdaguecyeoh: wrong url?12:24
ihrachyssdague: as per my understanding of _get_socket(), we get errno.EADDRINUSE from .listen()12:24
cyeohsdague: oh sorry I meant
ihrachyssdague: indeed, though since the test is deprecated and removed in master, we search for an easy fix :)12:25
*** salv-orlando_ has joined #openstack-infra12:25
sdagueihrachys: sure, this is supposed to be unit tests though, runnable on anyone's machine, so you need to find a free port12:25
sdaguenot just die when a port is already used12:25
ihrachyssdague: for now, flaper87 changed 9000 to 9001, but it's not elegant enough :)12:25
sdagueihrachys: right, that's just going to fail somewhere else12:25
sdaguecyeoh: more, I don't understand why you'd change that12:26
ihrachyssdague: it seems we see other failures with this port in other projects, see
uvirtbotLaunchpad bug 1285593 in openstack-ci "storyboard-webclient tests fail with "Port 9000 is already in use by another process"" [Medium,Triaged]12:26
sdagueit's not meant to be a variable, it's meant to be a default12:26
flaper87(note that the change to 9001 is just on stable/havana and that test doesn't exist in master anymore12:26
flaper87but yeah, I agree it's not the nicest fix12:27
* flaper87 wasn't aware this was failing elsewhere too12:27
sdagueflaper87: sure, honestly, binding to ports shouldn't be happening in unit tests anyway. That's a bad pattern that's grown up12:27
cyeohsdague: well its a default that gets used in a couple of places12:27
sdaguecyeoh: right, and in all those places you can override already12:27
ihrachysflaper87: it doesn't mean your fix shouldn't go now though :)12:27
flaper87sdague: agreed!12:27
cyeohsdague: yep, it just means that it requires overriding in a couple of places, rather than one ...12:27
ihrachysflaper87: but there's smth to consider. like the way we assign ports.12:28
cyeohsdague, but I can live without it12:28
sdaguecyeoh: it's pretty much just FLAT_INTERFACE12:28
ihrachyssdague: I'm for mocking also, but I leave it up to flaper87 :_12:28
*** salv-orlando has quit IRC12:28
*** salv-orlando_ is now known as salv-orlando12:28
cyeohsdague: and VLAN_INTERFACE?12:28
sdagueFLAT_INTERFACE and VLAN_INTERFACE shouldn't ever get used at the same time (you can set both if you care)12:28
cyeohright, devstack does12:29
ihrachyssdague: we already mock .listen() in other test cases in this file, and the test checks thread pool, not socket interface, so...12:29
cyeohsdague: anyway I'll just drop it - for my case I only care about FLAT_INTERFACE really12:29
sdaguemostly because there is already a way to do it, I don't think we should make another way to do it, which then confuses people12:30
sdagueespecially as the existing variables match nova.conf variables12:31
openstackgerritRomain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions Fix climate.json reported error
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency
*** jasondotstar has joined #openstack-infra12:32
*** adalbas has joined #openstack-infra12:37
*** _nadya__ has joined #openstack-infra12:39
*** jgallard has joined #openstack-infra12:41
anteayayjiang have you read
rustlebeelifeless: it's up and running again (reviewstats)12:46
*** yolanda has joined #openstack-infra12:47
*** yolanda has quit IRC12:48
*** yolanda has joined #openstack-infra12:49
*** salv-orlando has quit IRC12:49
*** vkozhukalov has quit IRC12:49
*** vkozhukalov1 has joined #openstack-infra12:50
*** jeckersb_gone is now known as jeckersb12:51
*** miqui has joined #openstack-infra12:51
*** yolanda has quit IRC12:52
openstackgerritRomain Chantereau proposed a change to openstack-infra/reviewstats: Format json text Updated lp_projects definitions
*** mriedem has joined #openstack-infra12:54
*** e0ne has joined #openstack-infra12:55
*** thomasem has joined #openstack-infra12:55
openstackgerritIvan Melnikov proposed a change to openstack-dev/hacking: Consider sys.real_prefix for _get_import_type
*** thuc_ has joined #openstack-infra12:58
*** e0ne has quit IRC12:59
openstackgerritafazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined
*** alexpilotti has quit IRC13:00
*** dprince has joined #openstack-infra13:01
*** bknudson has joined #openstack-infra13:06
*** yassine has quit IRC13:08
*** mwagner_lap has quit IRC13:09
*** e0ne has joined #openstack-infra13:11
*** matsuhas_ has quit IRC13:12
*** matsuhashi has joined #openstack-infra13:13
openstackgerritafazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined
*** matsuhashi has quit IRC13:17
*** julim has joined #openstack-infra13:18
*** yassine has joined #openstack-infra13:21
sdagueafazekas: if you are trying to fix trove on stable/havana, that's not the right way to do it13:23
*** jhesketh_ has quit IRC13:23
*** rustlebee is now known as russellb13:23
*** CaptTofu has quit IRC13:24
*** jamespage has quit IRC13:24
*** nosnos has quit IRC13:24
*** nosnos has joined #openstack-infra13:25
*** dcramer_ has joined #openstack-infra13:27
afazekassdague: What is purpose of the OVERRIDE_ZUUL_BRANCH variable ?13:29
SergeyLukjanovhey folks13:29
sdagueafazekas: a bunch of internals in zuul, don't touch those13:29
afazekassdague: where is the ZUUL_BRANCH defined normally ?13:29
SergeyLukjanovwhy we're not removing stable/havana from trove if it's not working?13:29
sdagueafazekas: zuul creates those13:29
sdagueSergeyLukjanov: exactly13:29
*** nosnos has quit IRC13:29
fungiafazekas: OVERRIDE_ZUUL_BRANCH is for when you want to force a specific branch to be used, rather than letting zuul specify it for you based on the branch the proposed commit appears in13:30
afazekaslooks like the trove should not be enabled service when the ZUUL_BRANCH is stable/havana13:31
sdagueafazekas: that's probably true13:31
sdagueso go into and change it there where we do service selection13:31
sdaguewhich is what I said in my first -1 in that review13:32
*** aysyd has quit IRC13:32
afazekassdague: the service selection seams good in the  devstack-vm-gate.sh13:33
*** vhoward has joined #openstack-infra13:34
sdagueafazekas: the value is suspicious13:34
sdaguedevstack have a bad default on stable/havana ?13:34
*** eharney has joined #openstack-infra13:35
afazekassdague: AFAIK the localrc overrides all default13:35
*** andreaf has joined #openstack-infra13:36
sdaguesure, so figure out where that's getting set13:37
afazekasFIXME:  True or False: If I made a change on devstack-gate it will use the 'master' as ZUUL_BRANCH even on the stable jobs13:38
*** jgrimm has quit IRC13:39
mordredmorning all13:39
mordredafazekas: true. and hrm.13:41
mordredsdague: we don't really have a job that runs master d-g against stable/havana do we?13:42
*** salv-orlando has joined #openstack-infra13:43
fungimordred: we do13:43
*** yolanda has joined #openstack-infra13:43
*** njd002 has quit IRC13:43
*** jasondotstar has quit IRC13:43
fungimordred: just not a periodic job13:43
fungiwe test proposed d-g changes against stable/havana13:44
*** dkliban has joined #openstack-infra13:44
*** markmcclain has joined #openstack-infra13:44
openstackgerritThierry Carrez proposed a change to openstack-infra/storyboard: Remove legacy priority column from stories table
*** wenlock has joined #openstack-infra13:47
mordredfungi: when you awaken, could you take a peek at ? it's ready to go13:48
mordredoh - fungi is here13:48
* mordred was in scrollback13:48
mordredfungi: where?13:48
mordredfungi: I was just looking through layout.yaml and didn't see anything13:48
mordredoh - duh. in the check queue13:49
*** thuc_ has quit IRC13:49
* mordred shuts up13:49
* mordred may not be awaken yet13:49
fungiyup. non-voting for fear that people wouldn't be able to merge d-g changes since stable is broken so often13:49
*** thuc has joined #openstack-infra13:49
openstackgerritA change was merged to openstack-infra/storyboard: Comments controller
*** jeckersb is now known as jeckersb_gone13:50
*** yolanda has quit IRC13:50
*** jepoy has joined #openstack-infra13:50
*** yolanda has joined #openstack-infra13:51
*** thuc_ has joined #openstack-infra13:51
fungimordred: does storyboard-db-manage load_projects rely on those projects being clonable via git? just making sure we're not introducing a new manage-projects race condition like we have with repo replication13:51
fungimordred: and by ready to go you mean that storyboard has actual db migrations now (per clarkb's earlier -1)?13:53
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Adds second hiera data directory
mordredfungi: it does not13:53
mordredfungi: yes -13:54
*** rcleere has quit IRC13:54
mordredfungi: (it does not depend on git things, and yes, it has migrations - I have run that script by hand on the storyboard server and it works)13:54
*** wchrisj has joined #openstack-infra13:54
*** thuc has quit IRC13:54
*** yamahata has joined #openstack-infra13:54
*** jeckersb_gone is now known as jeckersb13:54
mordredfungi: also, have you logged in to storyboard yet? I'm making a patch that adds superusers, and it's easier if you've logged in already13:54
mordredfungi: ah - I see that you have13:55
fungii think i havem but not sure if there have been any db wipes since. i'll log in again13:55
fungidone, though logging in gets me a completely empty page (i assume that's a known issue)13:55
*** jamespage has joined #openstack-infra13:56
*** rossella_s has quit IRC13:56
*** vkozhukalov1 has left #openstack-infra13:57
anteayasince we are in a soft freeze on puppet changes until merges, how far away are we from it merging?13:58
*** nosnos has joined #openstack-infra13:58
*** _nadya__ has quit IRC13:59
*** matsuhashi has joined #openstack-infra13:59
openstackgerritDolph Mathews proposed a change to openstack-infra/elastic-recheck: provide more specific feedback to query authors
openstackgerritA change was merged to openstack-infra/config: Load storyboard projects from projects.yaml
*** nosnos has quit IRC14:00
*** nosnos has joined #openstack-infra14:01
*** aysyd has joined #openstack-infra14:01
*** wenlock has quit IRC14:05
*** nosnos has quit IRC14:05
*** jepoy has quit IRC14:07
*** rlandy_ has joined #openstack-infra14:08
*** jepoy has joined #openstack-infra14:08
*** salv-orlando_ has joined #openstack-infra14:08
*** rossella_s has joined #openstack-infra14:08
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Load storyboard superusers from yaml file
*** aysyd has quit IRC14:09
*** kashyap has quit IRC14:10
*** jasondotstar has joined #openstack-infra14:10
*** salv-orlando has quit IRC14:10
*** salv-orlando_ is now known as salv-orlando14:10
*** jp_at_hp has quit IRC14:10
*** CaptTofu has joined #openstack-infra14:10
*** nosnos has joined #openstack-infra14:11
*** rlandy has quit IRC14:11
*** thuc_ has quit IRC14:12
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Just make popularity-contest absent
*** skraynev is now known as skraynev_afk14:12
*** jepoy has quit IRC14:12
*** thuc has joined #openstack-infra14:12
openstackgerritThierry Carrez proposed a change to openstack-infra/storyboard: Remove legacy priority column from stories table
*** nosnos has quit IRC14:14
*** krtaylor has quit IRC14:15
*** adalbas has quit IRC14:16
*** matsuhashi has quit IRC14:16
*** jp_at_hp has joined #openstack-infra14:16
*** rwsu has joined #openstack-infra14:17
*** thuc has quit IRC14:17
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better
mordredfungi: ^^ that fixes your comment - and thanks, you're right, we do not need to do that by hand anymore14:17
fungimordred: lgtm14:19
*** mfer has joined #openstack-infra14:19
openstackgerritmark mcclain proposed a change to openstack-infra/config: Create neutron-specs git repo
mordrednibalizer: at some point, not today, we should investigate whether we can merge with the stankevitch-python module you used for puppetboard14:20
*** jgrimm has joined #openstack-infra14:21
*** aysyd has joined #openstack-infra14:22
mordredsince that module provides pip, it seems like perhaps adding a few more flags to encompass installing latest pip and/or describing a thing for pip3 would be nice-ish14:22
*** thuc has joined #openstack-infra14:23
*** yassine has quit IRC14:25
*** rcleere has joined #openstack-infra14:27
*** aysyd has quit IRC14:27
mordredanteaya: I think as soon as either SergeyLukjanov, clarkb or jeblair get a chance to do a second pass on it, we can land it and then watch any carnage14:27
* anteaya gets ready for carnage14:28
SergeyLukjanovmordred, re pip?14:28
anteayafortunately the new gui should give us all the gory details14:28
*** vkozhukalov1 has joined #openstack-infra14:29
jeblairmordred: +2; are you going to try some manual applications of that?14:31
*** adalbas has joined #openstack-infra14:31
fungii'd like to at least make sure it's been tried out selectively on *a* server whjich gets things from pip before we punch it through14:31
fungier, what jeblair said14:31
SergeyLukjanovmordred, +2 from me too for 5142514:31
*** matsuhashi has joined #openstack-infra14:32
fungiif we know it completes on one of our servers, then we can have some confidence that whatever carnage there is will hopefully not be widespread14:32
openstackgerritA change was merged to openstack-infra/config: Just make popularity-contest absent
fungiwe'll probably also want to try manual image updates of each basic puppeted slave class to make sure there's nothing obviously wrong (or to be able to catch it early if there is, since we're not going to see that via puppetboard)14:33
*** DuncanT- has quit IRC14:36
*** yassine has joined #openstack-infra14:37
*** marktraceur has quit IRC14:38
*** thuc_ has joined #openstack-infra14:39
SergeyLukjanovfolks, let's make noop real! any objections?14:39
*** saju_m has quit IRC14:39
*** homeless has joined #openstack-infra14:40
*** marktraceur has joined #openstack-infra14:41
jeblairSergeyLukjanov: no objections!14:41
* SergeyLukjanov breaking the world!14:41
SergeyLukjanovooops, I mean making noop real :)14:41
*** maxbit has joined #openstack-infra14:41
fungii take it the functionality for that is already merged and zuul's been restarted?14:42
jeblairfungi: yes i did that yesterday14:42
*** thuc has quit IRC14:42
fungiawesome. i didn't pay close enough attention to scrollback14:43
jeblairfungi: i also restarted the mergers, so they should have their init scripts and auto-re-registration bugs fixed14:43
*** hub_cap_ has quit IRC14:43
SergeyLukjanovjeblair, cool14:43
*** thuc_ has quit IRC14:43
*** david-lyle has joined #openstack-infra14:43
*** mwagner_lap has joined #openstack-infra14:43
*** krtaylor has joined #openstack-infra14:43
fungijeblair: excellent, so next zuul restart we just restart the daemon on zuul.o.o and everything should simply work, right?14:44
jeblairfungi: yep14:44
jeblairwhile we're breaking things, i also approved the revoke more sudo patch14:44
*** zns has joined #openstack-infra14:45
pabelangermorning! Is there a list of jenkins plugins that nodepool needs / uses?  I couldn't see anything obvious14:45
fungipabelanger: the zeromq event publisher14:45
*** marktraceur has quit IRC14:46
pabelangerfungi, Yup, have that one. Looks to be the only one right?14:46
fungipabelanger: as far as i know, yes14:46
pabelangerokay cool14:46
*** marktraceur has joined #openstack-infra14:47
*** dims has quit IRC14:47
anteayaSergeyLukjanov: could you state your reasons for your -1 on on the patch?14:47
anteayaso that markmcclain knows what he needs to chage14:48
*** thedodd has joined #openstack-infra14:48
fungipabelanger: yep, that's all we mentioned in so i think you should be set14:48
fungianteaya: he did in his comment immediately prior to the -1 vote14:48
fungianteaya: looks like he simply forgot to set the vote on the previous comment and was adding it with a follow-up empty comment14:49
SergeyLukjanovanteaya, we've decided to set topic for new-projects-creation CRs 'new-project'14:49
anteayaSergeyLukjanov: yeah, didn't know that was the cause of the -114:49
anteayaokay thanks14:49
afazekasjeblair: looks like it changed all stable jobs to SUCCESS on the devstack-gate, what is the normal way to fix it ?14:50
SergeyLukjanovanteaya, it could be 0, but it's harder to identify later which CRs has been reviewed...14:50
*** thuc has joined #openstack-infra14:50
fungialso, i guess the nova experiment which is only just underway to try doing blueprints in a git repo is creating an avalanche of other programs wanting to try the same experiment without first waiting to see if it's a success?14:50
jeblairafazekas: did you look into sdague's suggestion?14:50
*** atiwari has joined #openstack-infra14:51
pabelangerfungi, great thanks. Wanted to check since I'm hitting an exception when nodepool tries to configure jenkins for a new build executor, just making sure no extra plugin was needed14:51
afazekasjeblair: the service adds are correct14:51
openstackgerritmark mcclain proposed a change to openstack-infra/config: Create neutron-specs git repo
afazekasjeblair: the problem is it behaves like it were a master job even when it is stable/havana14:52
anteayaSergeyLukjanov: sure, that makes sense14:52
mordredjeblair: yes. I will now run it manually on a server or two before doing the +A14:52
jeblairafazekas: is this only on periodic jobs?  (eg, jobs for changes on the branch are correct, but periodic jobs are incorrect?)14:53
mordredjeblair: how about graphite? that's heavy-python but also if it borks for a second because we missed something won't kill the world14:53
afazekasjeblair: it is a not voting job on the devstack-gate14:53
jeblairmordred: wfm14:53
afazekasjeblair: You can check the jenkins response on my change and on another one for example:
jeblairafazekas: what is not a voting job?  i'm trying to understand in what conditions it does not work as expected14:54
openstackgerritA change was merged to openstack-infra/config: Remove Oneiric conditional logic
*** wchrisj has quit IRC14:54
afazekascheck-tempest-dsvm-postgres-full-havana FAILURE in 8m 27s (non-voting)14:55
afazekasvs check-tempest-dsvm-postgres-full-havana SUCCESS in 38m 04s (non-voting)14:55
pabelangerfungi, is it possible I need: ?14:56
pabelangerI _think_ I do14:56
jeblairafazekas: right, so it's the branch override jobs (which are mostly periodic, but also some of the cross-branch testing like the one you're pointing out)14:57
*** salv-orlando_ has joined #openstack-infra14:57
jeblairafazekas: so your change may be correct, but i didn't understand what you were fixing from the commit message14:57
jeblairafazekas: would you mind updating that a bit, and also adding a test for it in
*** salv-orlando has quit IRC14:58
*** salv-orlando_ is now known as salv-orlando14:58
nibalizermordred: definitely!!14:58
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Authorization models added
jeblairafazekas: oh, hrm, we don't have any setup_localrc tests... in that case, it may not be easy to test14:58
openstackgerritA change was merged to openstack-infra/config: Revoke sudo from almost all jobs
jeblairafazekas: at any rate, maybe you could at least update the commit message with more of an explanation?14:59
jeblairpabelanger: seems likely14:59
*** dims has joined #openstack-infra15:00
fungipabelanger: i think so, yes15:00
*** pcrews has joined #openstack-infra15:01
sdaguejeblair: do you have the log size / day current growth rate? I'd like to understand how bad we are at the moment, so how aggressively we need to dump things15:01
*** vkozhukalov1 has left #openstack-infra15:03
pabelangerfungi, jeblair: okay, testing the theory now.15:03
sdaguefungi: are we deleting old logs regularly as well?15:03
fungisdague: yes, any which are >180 days15:03
fungisdague: and we also currently run a weekly pass to compress anything compressable which was not already15:04
sdagueok, so the raw total is a little less useful than delta / week (once everything is accounted for, compression + deletes)15:04
jeblairclarkb: when you have a second, the thread "[Openstack-i18n] Horizon translations - and update of strings" has a question for you15:05
sdagueis there a way to get that?15:05
*** thuc_ has joined #openstack-infra15:05
jeblairsdague: i would just zoom in and subtract15:05
*** dhellmann_ is now known as dhellmann15:05
jeblairsdague: (if you click-drag on cacti graphs they zoom)15:05
*** mgagne has joined #openstack-infra15:05
jeblairsdague: (zoom in to get one day and then read "current" value)15:06
afazekasjeblair: ok15:06
yolandahi jeblair, do you have a second? i wanted some feedback from that bug:
uvirtbotLaunchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed]15:07
*** salv-orlando has quit IRC15:07
*** thuc has quit IRC15:08
*** vkozhukalov has joined #openstack-infra15:08
*** dangers_` is now known as dangers15:08
*** salv-orlando has joined #openstack-infra15:09
*** todd_dsm has quit IRC15:11
*** SumitNaiksatam has quit IRC15:11
*** vkozhukalov has quit IRC15:12
openstackgerritafazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH as ZUUL_BRANCH when defined
*** zhiyan is now known as zhiyan_15:13
jeblairyolanda: i think the idea was that if someone changed a requirement (instead of adding or removing one), then the script should try to make sure the old and new values overlapped15:14
jeblairyolanda: i don't know for certain if that's possible or how to do that, but it's worth looking into the api15:15
yolandajeblair, i started looking at that scripts:
yolandaso is ok to look at what this job is doing and dig more on it?15:16
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better
*** marktraceur has quit IRC15:16
*** wenlock has joined #openstack-infra15:16
jeblairyolanda: we were thinking about this job:
*** mwagner_lap has quit IRC15:17
*** jswarren has joined #openstack-infra15:18
*** marktraceur has joined #openstack-infra15:18
yolandaok, that script /usr/local/jenkins/slave_scripts/project-requirements-change.py15:18
*** salv-orlando has quit IRC15:19
*** beekneemech is now known as bnemec15:20
*** prad has joined #openstack-infra15:20
*** prad has left #openstack-infra15:20
*** prad has joined #openstack-infra15:21
*** salv-orlando has joined #openstack-infra15:21
jeblairyolanda: right15:21
yolandaok, let me take a look at it15:21
*** AJaeger has joined #openstack-infra15:22
AJaegerHi infra team, I would appreciate a review of the following patch to better handle the translation jobs:
*** matsuhashi has quit IRC15:25
jeblairfungi, clarkb, mordred: glance -d --os-image-url= --os-image-api-version=2 image-list15:25
*** thuc_ has quit IRC15:25
jeblairfungi, clarkb, mordred: that works in rax ^15:25
mordredjeblair: w00t15:25
*** nati_ueno has joined #openstack-infra15:26
jeblairfungi, clarkb, mordred: the first arg is to get around a bug in rackspace's non-keystone where it spits out the wrong endpoint url... or something like that15:26
*** thuc has joined #openstack-infra15:26
* mordred cries15:26
jeblairfungi, clarkb, mordred: the second is that for some reason, we have to force the v2 endpoint... i don't know if that's because of the first bug or what.15:26
mordredjeblair: I'm assuming we can probably set OS_IMAGE_URL ish15:26
jeblairmordred: yeah15:26
mordredjeblair: so now I suppose we've got some motiviation to move off of hpcloud 1.0 completely15:27
*** mwagner_lap has joined #openstack-infra15:27
*** todd_dsm has joined #openstack-infra15:27
jeblairSergeyLukjanov: oh no, noop failed (i think it needs to be updated because a new gate-noop job slipped in)15:28
jeblairmordred: i have a patch series up for that15:28
*** matsuhas_ has joined #openstack-infra15:28
jeblairfungi: yeah, so about -specs repos...15:28
jeblairi'm pretty sure the mailing list posts were about "doing an experiment"15:28
fungii fear we're facing an explosion of -specs repos which will get abandoned soon15:29
jeblairnot "changing the way all the projects handle enhancement proposals in an ad-hoc and inconsistent manner"15:29
fungiand then we get to cart them around indefinitely15:29
*** packet has joined #openstack-infra15:29
pabelangerfungi, do you have Global credentials setup for SSH statically, or is nodepool adding them each time a new node comes online.  If I understand correctly, nodepool expect the Global credentials to be already setup?15:30
jeblairi'd personally rather avoid that...15:30
*** thuc has quit IRC15:31
jeblairwe've already seen that nova and qa specs repos don't plan on following the same standards15:31
fungipabelanger: nodepool's config had the uuid of the jenkins ssh credential from the master's config.xml15:31
jeblairso i'm really concerned about the experience for contributors15:32
mordredjeblair: the patch does not work - and on further inspection, this is not the same as the one we're running successfully inside of HP15:32
jeblairmordred, ttx: thoughts on specs repos?15:32
jeblairmordred: the pip patch?15:32
mordredjeblair: so, consider soft-freeze lifted until futher notice - this will require work15:32
mordredjeblair: yes15:32
* mordred grumbles things about ruby15:32
jeblairmordred: can you wip that?15:32
mordredjeblair: yes15:32
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Make pip install from upstream better
mordredpushing up latest edits first15:33
*** matsuhas_ has quit IRC15:33
*** jpeeler has joined #openstack-infra15:33
mordredjeblair: so- specs repos - I'd like to stop creating any more15:33
mordredjeblair: I'm a bit annoyed that we have 2 divergent options already15:34
mordredjeblair: why are they not folloing the same standards?15:34
jeblairmordred: because they need "freedom to experiment"15:34
mordredI mean, of all the projects, shouldn't nova and qa know the value of collaboration?15:34
mordredjeblair: bah. they need freedom to be lazy and not talk15:35
mordredthat's bullshit15:35
markmcclainmordred: so we're aligning with Nova here… what do you suggest?15:36
mordredsdague: ^^ I believe I'm calling you out ina  rude manner15:36
pabelangerfungi, Great thanks. I think that was the last piece I was needing:
mordredmarkmcclain: I suggest waiting until we have more agreement on what the heck is going on, tbh15:36
ttxjeblair: yes, i see it as an experiment15:36
fungisince storyboard is aimed at solving the blueprint situation, it would be nice to at least have the current experiment limited to figuring out what new blueprint workflow options storyboard should be considering15:36
mordredthis is quickly escalating from "nova is experimenting with something new"15:36
ttxfrankly, it looks all shiny but I want to see how it works in the wild15:37
mordredto "ZOMG PONIES!!! quick, everyone adopt the not-yet-finished-experiment"15:37
mordredmind you - I _LIKE_ the idea15:37
ttxbecause in the past, all attempts to add more process weas met with skepticism15:37
fungittx: well, we've now got several projects wanting to ditch lp blueprints in favor of whatever nova's experiment is15:37
ttxfungi: they don't ditch Lp15:37
fungiwhich feels premature to me15:37
sdaguemordred: because I actually specifically want to try somethings differently to figure out what works and what doesn't15:38
ttxfungi: they just use an out-of-LP approval process15:38
mordredsdague: ok. you see what's happening though15:38
markmcclainfungi: the problem with lp is the process for commenting on, revising and versioning a blueprint spec is non-existent15:38
sdaguebecause I think coming up with standardized process right now, with no real experience, just means we'll pick process before knowing what works15:38
*** wchrisj has joined #openstack-infra15:38
mordredsdague: how about we keep the experiment to one project15:39
mordredand try different things15:39
*** gyee has joined #openstack-infra15:39
jeblairmordred: indeed, that's usually how we experiment -- we let someone trailblaze, try a couple different paths, and then see what worked15:39
*** marktraceur has quit IRC15:39
fungimarkmcclain: i understand it's broken, just worried that if nova's "approving blueprints through code review" experiment is aborted, we end up with neutron, qa and possibly a dozen other programs who have decided to follow suite which now have dead specs repos hanging around15:39
*** CaptTofu has quit IRC15:39
sdaguemordred: because the current process is so broken on qa front, we basically told people to stop creating blueprints entirely15:39
ttxIn the past, the idea would have been rejected fast, but it sounds like we've reached a critical mass of processs wonks / maturity15:39
sdaguethat's our current approach15:39
jeblairit's much easier to get people on board with the final outcome than having 20 experiments, all of which come up with something different, and all of which people think are successful15:40
mordredsdague: right. we did the same thing here15:40
sdagueand it sucks15:40
*** CaptTofu has joined #openstack-infra15:40
mordredsdague: yes. I'm talking about two different groups expermenting independently15:40
ttxfungi: so the issue is... we don't know how to delete repositories15:40
mordredI think the idea of blueprint dev in code review is awesome15:40
fungiand what is the end-game we envision for these specs repos when storyboard blueprints are up and running?15:40
fungittx: the issue is that deleting repositories means erasing history15:40
*** marktraceur has joined #openstack-infra15:41
ttxfungi: as I commented on the ML, storyboard will be happy having -specs tasks15:41
sdaguebut a blueprint for nova looks radically different than a blueprint in qa program, by definition15:41
fungieven if we get the technical hurdles down15:41
mordredsdague: I disagre with teh "by definition"15:41
mordredthat is an assertion you are making, I do not think that it is necessarily true15:41
sdagueso if you really want we can punt on this entirely with qa, and just go completely out of process and use google docs are something15:41
fungittx: ahh, i'm back to missing context obviously--been too busy to read e-mail again. i'll just lurk this conversation15:41
sdaguewhich is basically waht people are doing now15:41
mordredsdague: no - I'm asking you to help us try to keep the project from balkanizing15:42
ttxI'd prefer the experimentation to be limited before it is successful for at least one project15:42
mordredwhich is a hard enough task without introducing a billion divergent mechanisms for blueprint development15:42
sdaguemordred: we already have a billion divergent mechanisms for blueprint development15:43
mordredsdague: and you don't consider tah a bug?15:43
markmcclainthe only problem of letting one project experiment is the others who aren't chosen keep suffering from the chaos we have now15:43
mordredwhy would we CODIFY that inside of actual openstack systems?15:43
ttxmarkmcclain: they admittedly had the idea first :)15:43
sdagueI do, and using a common tool, that lets me purge random etherpad, wiki, google docs, regardless in not being a perfect clone of nova, is a real step forward15:43
markmcclainttx: agreed but it is a good one15:44
*** andreaf2 has joined #openstack-infra15:44
mordredso - what is it that is divergent in the nova and qa approaches?15:44
jeblairmordred: directory structure, document types...15:44
sdaguethe document types aren't divergent15:45
*** andreaf has quit IRC15:45
sdaguedirectory structure is15:45
*** CaptTofu has quit IRC15:45
jeblairi suggested that they should be sphinx docs, and we can generate actual sphinx docs from them15:45
jeblairbut i could not get them to agree on that15:45
mordredjeblair: wow, why not? that seems obvious to me15:45
sdaguethey are both rst docs15:45
mordredok - looking at nova, I'd liek to make some suggestions15:46
mordred"approved" is duplicative15:46
mordredthe spec is approved when it lands in the repo15:46
mordred"implemeneted" is duplicative - status of a blueprint is tracked currently in blueprints15:46
jeblairmordred: i was hoping we could plug them into the project testing interface even, so we wouldn't have to build new frameworks15:46
mordredjeblair: that would be neat15:46
mordredit's almost like we alraedy have tools for that15:47
sdaguewe don't have tools for the style checker I hacked out15:47
mordredand why would there be a juno dir and not a branch?15:47
*** dkliban_ has joined #openstack-infra15:48
mordredsdague: I'm sure you can run a style checker on things that fit into the current publication toolchain15:48
mordredI'm sorry - the idea of specs in git repos is great - but this is a mess15:48
*** dkliban has quit IRC15:49
jeblairi don't see how we can say it's an experiment, and yet have all the projects doing it15:49
jeblairwe need to either keep it small so that we actually know what we're experimenting with15:49
jeblairor it needs to grow up quick so we can scale it out to all the projects15:49
ttxMy main issue with the idea is that the whole thing fails when what you want to discuss is cross-project15:51
sdagueso I get that all the ducks are in a row at this point15:51
ttxwould you then have parallel -specs change in various repos ?15:51
sdaguebut I don't think people realize how truly terrible the current situation isu15:51
jeblairsdague: i do15:51
ttxbut I'm happy to trat cross-project as an exception procedure15:51
sdagueand that even divergent specs repos is a giant step forward15:52
sdaguein project commonality15:52
sdaguejeblair: so that's what I'm trying to reconcile, because I don't understand how this can be seen as a bad thing given the current state15:53
*** BadCub01 has joined #openstack-infra15:53
sdaguettx: there is 0 ability to do cross project blueprints today, so this doesn't make that any worse15:54
*** thuc has joined #openstack-infra15:55
ttxsdague: agreed. I was just discussing using that system as the long-term solution15:55
*** marktraceur has quit IRC15:55
*** marktraceur has joined #openstack-infra15:55
ttxsdague: basically I agree it's a temporary improvement. the problem the -infra team has is in the "temporary" aspect15:55
sdagueit's an incremental improvement15:56
ttxsince creating repositories is not really "temporary"15:56
sdaguethe only way we get anywhere is incrementally15:56
ttxsdague: wouldn't cause a problem if we had more agility around repositories creation/deletion/renaming15:57
mordredsdague: right - the problem isn't that you're working on it, nor that the attempts aren't perfect yet15:57
jeblairsdague: the openstack project, as a development organization, scales by sharing process; every -spec repo that's different is more confusion and more work15:57
sdaguettx: so can we then solve that problem instead?15:57
jeblairttx: i disagree about deleting -- even if we could, i don't think we would ever want to delete anything15:57
mordredsdague: the problem we're worried about is what jeblair just said15:57
sdaguejeblair: and we'll get there, but we aren't sharing process today15:57
sdaguewe really aren't15:57
sdaguemaybe it's not visible so clearly until we get here15:58
mordredsdague: right. but encoding divergent processes here is what troubles us15:58
sdaguebut go try and propose and get approved a blueprint across heat, nova, neutron, qa15:58
jeblairyou can even see it in our natural tendency -- two projects adopted this thing, so other projects start doing it, because so many people know the value of that consistency15:58
sdaguemordred: it's not encoding it, it's let us evolve it. And trusting that people will actually try to diverge over time15:59
jeblair(we'll have stackforge repos wanting them soon to, no doubt)15:59
mordredyeah - I think what we're probably saying is - perhaps experimentation is not as important as just rolling is out15:59
*** thuc_ has joined #openstack-infra15:59
mordredor, what i'm probably saying - I don't want to speak for others15:59
sdaguesorryr - And trusting that people will actually try to *converge( over time15:59
*** blamar has joined #openstack-infra15:59
*** blamar has quit IRC15:59
jeblairsdague: and there i was thinking i had won you over.  :)15:59
*** blamar has joined #openstack-infra16:00
mordredsdague: so, why don't we all just put our heads together, cook up one thing that's probably less bad than current, and then let the tidal wave of people who want to start using it start using it16:00
mordredsdague: and if we discover places it sucks, we can fix taht everywhere at once?16:00
*** michchap has quit IRC16:01
sdaguebecause then we end up in oslo gridlock16:01
jeblairi really do want storyboard to handle this; i don't think this should be necessary in the long term, so if we make a lot of them, we really are going to need openstack-attic/16:01
mordredoslo gridlock is caused by allowing divergence16:01
sdaguemordred: nope16:01
sdagueno, it really isn't16:01
*** thuc has quit IRC16:01
* jeblair registers openstack-attic/ on github16:01
*** dkliban_ is now known as dkliban16:01
*** jnoller has joined #openstack-infra16:01
sdaguego look at the current state of oslo sync trees in projects16:02
mordredthat's what I mean16:02
sdaguethere is massive divergence16:02
mordredbecause we allowed it because "the apis might not be solild yet"16:02
sdaguebecause process requires getting everyone to agree to something16:02
mordredso we started copying stuff16:02
mordredexcept at different cadences16:02
mordredas opposed to eating it and using libraries from the start16:02
mordredwhcih has a known mechanism for sharing16:02
mordredoslo-incubator hell is EXACTLY what I'm worried about with divergent spec repo approaches in the name of "freedom to experiment"16:03
*** reed has joined #openstack-infra16:04
*** afazekas_ has joined #openstack-infra16:04
sdagueso we can either have a bike shedding up front about if you encode releases / states in directory names. Or we can try it a couple different ways. Then say "oh, that sucked when we had to do all those git moves" and give up on it16:04
mordredif we make one spec repo format and a set of tools that work with it, then iterating on that involves iterating on the one set of tools, and we can keep people in somewhat of a similar place16:04
mordredI would prefer the bikeshedding up front16:04
mordredit's called "design"16:04
mordredand it's actually not bikeshedding - it's important16:05
sdaguebut design without experience ends up going around in circles. I feel like we'd have a really useful design session on common format after a release and experiences16:05
fungialso, we won't be just be trying it a couple different ways if every program decides they want to jump on board while the one or two which are piloting the workflow haven't arrived at any recommendations yet16:05
mordredthat's the problem16:06
sdaguewe'll be doing what we did with hacking.py16:06
mordredit's not going to be a couple16:06
sdaguewhere we copy it around between like minded projects for a while16:06
*** afazekas has quit IRC16:06
sdaguethen figure out what's working common and extract16:06
mordredthat's not what we do16:06
sdagueanyway, that was my thinking16:06
mordredwith hacking16:06
sdaguemordred: that's exactly what we did with hacking16:06
mordredthe instant I saw a copy thing go into oslo16:06
mordredI made a library16:06
sdaguewe were copying it around in tempest for a whole release prior to that16:07
sdagueand back and forth fixes to nova16:07
mordredI just didn't see that :)16:07
mordredbut this isn't code16:07
mordredthis is process that involves tooling and repos16:07
mordred"just copy the file" doesn't have a good analogue here16:07
*** sahumada has quit IRC16:08
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Replace gate-noop with noop
jeblair16:07 < openstackgerrit> Matthew Treinish proposed a change to openstack/qa-specs: Add sphinx support to qa-specs repo
jeblairrelevant ^16:09
mordredyup. ESPECIALLY since it is a patch that's not in line with how our sphinx tooling works16:09
jeblairoh!  i hadn't actually looked at it16:10
jeblairmtreinish: ^16:10
mordredmtreinish: we know how to build sphinx things - and publish them16:10
jeblairyeah, that would need a tox.ini16:10
*** asselin has joined #openstack-infra16:14
jeblairso i'm going to reply to the -dev list and ask other projects to hold off at least a little bit so we can try to get some really basic stuff sorted out first16:14
sdaguejeblair: it's probably worth getting deep into the threads on the nova specs repo than as well16:15
*** ominakov has quit IRC16:15
jeblairi don't think we have consensus on whether or not we should keep to the current repos for a whole cycle, or open the floodgates, but i do feel pretty strongly that if we're going to open the floodgates, we need to have a few basic things worked out16:16
sdaguebecause the problem is this current need by projects is completely unaddressed in anything we have currently, so it doesn't surprise me that people are jumping at anything that might make their life better16:16
sdaguejeblair: ok, so can you nail down what you think those things should be in a thread16:16
reedhello folks16:16
sdagueso we can at least try to agree on the things that we think have to be identical between projects, and the places where they can adapt to local needs16:17
markmcclainsorry resuming the conversation, but the other important item is I'd prefer not the spend a long time iterating on this as folks are already writing specs for Juno16:17
mordredsdague: that would be great. I think that we can probably get a common scaffolding in place withouth years of bikeshedding16:17
nibalizermordred: left you a short essay on the 'make pip better' change16:17
mordredthat still allows for exploration per-project16:18
*** michchap has joined #openstack-infra16:18
*** mihgen has quit IRC16:18
*** sdake_ has joined #openstack-infra16:18
*** sdake_ has quit IRC16:18
*** sdake_ has joined #openstack-infra16:18
jeblairsdague: yeah.  i think they are a) sphinx yes/no? b) testing/publishing interface c) what information is encoded (don't duplicate blueprint status) d) workflow process (how it interacts with blueprints)16:18
mordredI agree with that list16:19
sdaguejeblair: sure, so get that out in a thread, especially if it contradicts what's in nova-specs today16:19
mordredtbh - spec template format divergence sounds like a fine place to play16:19
sdagueI honestly think directory structure is probably going to vary as well16:20
sdagueas keeping track of things with 150 active blueprints is different than with 2016:20
anteayaso should we ask markmcclain to WIP for the moment?16:20
fungimaybe approach directories vs branches specifically as an item to be ironed out16:20
sdaguewhich is why the qa-specs repo didn't put release in the directory structure, and had an implemented, largely as an attic16:21
mordredbut I think that we can have a structure where you can have arbitrary directory structure that still has a common entry point16:21
zarofungi: help with publication16:21
fungizaro: what's up?16:21
zarofungi: so i modified the 'overview' slides for gerrit uc.  would like to put that up on publications but not sure how i should do that.16:22
zarofungi: i don't think i want to make a branch for it.16:22
zarofungi: because it's just a modified version of overview.16:22
fungizaro: is it an improvement to the current presentation, or a separate presentation?16:23
mordredsdague: sure - but adding a release dir probably wouldn't kill you if it that was just what folks felt was necessary16:23
zarofungi: it's a shortened version of it, not really improvement.16:23
sdaguemordred: tempest doesn't have releases16:23
zarofungi: it was only 30 min talk.16:23
sdaguenot in the same way as server projects16:23
fungizaro: mmm, that seems like fodder for a separate branch (an abbreviated version of the talk?)16:24
sdagueand the qa-specs repo was going to be bigger than tempest anyway as it was going to be program level16:24
mordredsdague: sure. but you live in the release cycle, and you do tend to have divergence per release16:24
mordredlike, temptest for icehouse and tempest for juno will be different things, no?16:24
sdaguemordred: in all fairness the only reason for the tag is because the APIs are too unstable16:25
sdagueotherwise it's just tempest, that works against openstack apis16:25
mordredsure. but, well, let me put it a different way - do you not have a set of things you plan to work on in the juno cycle?16:25
mordredthat you plan to discuss some of at the juno design summit?16:25
zarofungi: it's actually abreviated but some added stuff as well.  maybe what i'll do is just add the stuff i added (multimaster jenkins) to the slides in the overview branch.16:26
sdaguebut I thought we were not trying to encode the things that blueprints provide16:26
sdaguelike targets16:26
sdaguebeing my logic16:26
fungizaro: we can branch the latest state of the overview branch and then you can propose the edits for the abbreviated talk you gave, then we approve and tag that for the date/venue16:26
mordredright. I don't think there should be a juno dir16:26
mordredbut - if, for some reason, it's needed everywhere else16:26
mordredthen it would not destroy qa to do it too16:27
mordredis all I'm saying16:27
jeblairzaro, fungi: or if it's a one-off, you can just push it up for review, and then one of us can tag that review as a tagged version of the overview talk as given16:27
fungizaro: because someone else might also want to give an abbreviated version of the talk later, which allows that to be a good starting point16:27
jeblair(either way)16:27
fungijeblair: sure, just a tag without really landing the commit on a particular branch. that could also work16:28
zarosounds fine.  should i mention that it should only be a tag in the commit message?16:29
fungizaro: sure, that would help16:29
*** bhuvan has joined #openstack-infra16:31
jeblairsdague, mordred, ttx: msg sent to openstack-dev16:33
jeblairrussellb: ^16:34
*** nati_ueno has quit IRC16:35
jeblairhashar: are you or someone else at wmf interested in reviewing zuul status page javascript changes?16:35
jeblairhashar: jhesketh is working on merging our status page changes back into zuul and getting the zuul version in shape so that we can use it (with our styling, etc)16:36
mtreinishmordred: oops yeah I forgot the tox.ini16:36
openstackgerritRomain Chantereau proposed a change to openstack-infra/reviewstats: Format JSON text
jeblairhashar: (i don't mean put our style in the zuul repo, i mean apply our style on top of what's there)16:37
hasharjeblair: I can ask our jQuery evangelist :-]16:39
hasharhe rewrote the status page to be more javascript like16:39
*** jasondotstar has quit IRC16:39
hasharjeblair: passing the message to him :-]16:40
jeblairhashar: great, thanks!16:40
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: WIP Refactor kibana module to add kibana3
hasharjeblair: mail pushed (he is not online right now). Thanks for the notification!16:43
*** nicedice has joined #openstack-infra16:43
jeblairhashar: thank you!16:43
hasharjust in time since I need to rush back home now :]16:44
mtreinishmordred: oh and the build dir path doesn't match what's expected in the run-docs slave script16:44
openstackgerritAkihiro Motoki proposed a change to openstack-infra/elastic-recheck: Add query for bug 1296825
uvirtbotLaunchpad bug 1296825 in openstack-ci "ERROR: InvocationError: '/home/jenkins/workspace/gate-neutron-python26/.tox/py26/bin/python" [Undecided,New]
*** packet has quit IRC16:46
*** hashar has quit IRC16:47
*** dizquierdo has quit IRC16:49
*** harlowja has joined #openstack-infra16:49
mordredmtreinish: you want me to lend a hand? or you got it?16:50
mtreinishmordred: I think I've got it16:50
*** jlibosva has quit IRC16:52
*** pblaho has quit IRC16:52
mordredmtreinish: cool. let me know - happy to help!16:52
*** prad has quit IRC16:52
*** jpich has quit IRC16:53
*** marktraceur has quit IRC16:53
mtreinishmordred:  I think got everything. Did I miss anything on this revision?16:55
*** mgagne has quit IRC16:56
*** hogepodge has joined #openstack-infra16:56
*** amcrn has joined #openstack-infra16:57
*** marktraceur has joined #openstack-infra16:58
*** sc68cal_ is now known as sc68cal16:59
*** mgagne has joined #openstack-infra16:59
*** mgagne has quit IRC16:59
*** mgagne has joined #openstack-infra16:59
*** maxbit has quit IRC16:59
*** vkozhukalov has joined #openstack-infra17:01
*** zns has quit IRC17:01
*** zns has joined #openstack-infra17:02
anteayadoes the wikipage: need a face lift under end results needed?17:02
*** kiall_ is now known as Kiall17:02
anteayalike mentioning py3?17:02
*** todd_dsm has quit IRC17:03
mordredmtreinish: looks good - have you tested that it makes the doc/build dir if there is no doc dir? also, you could probably leave teh version field out of setup.cfg17:03
mordredbut it's also fine17:03
mtreinishthat's a good idea, I probably should test it :)17:05
fungimordred: did come to your attention yet?17:05
uvirtbotLaunchpad bug 1296200 in pbr "Unit tests are failing in Sid" [Undecided,New]17:05
*** roeyc has quit IRC17:05
mtreinishseems to work fine17:05
*** markmcclain has quit IRC17:09
*** markmcclain has joined #openstack-infra17:09
*** pbelanyi has quit IRC17:09
*** jpich has joined #openstack-infra17:10
*** dprince has quit IRC17:11
fungiAlex_Gaynor: any news on getting updated with a workaround/solution to the empty __pycache__ dirs breaking pip problem?17:12
clarkbwow capslock17:12
SergeyLukjanovclarkb, morning :)17:12
jeblairclarkb: HAHA17:12
derekhDoes nodepool have logs that I can look at? looks like its having trouble building the template for tripleo-f20 want to try and figure out why?17:12
* clarkb fixes capslock on this machine17:12
jeblairderekh: yes!
derekhfungi: jeblair thanks17:12
*** dstufft has quit IRC17:13
clarkbjeblair: looking at the horizon thread now17:14
*** rlandy_ has quit IRC17:14
*** e0ne has quit IRC17:14
*** prad has joined #openstack-infra17:14
*** dstufft has joined #openstack-infra17:15
*** vhoward has left #openstack-infra17:16
*** thuc_ has quit IRC17:16
openstackgerritJason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions
*** nati_ueno has joined #openstack-infra17:17
*** thuc has joined #openstack-infra17:17
*** zehicle_at_dell has quit IRC17:19
*** melwitt has joined #openstack-infra17:19
*** wenlock_ has joined #openstack-infra17:21
*** wenlock has quit IRC17:21
*** yassine has quit IRC17:21
*** wenlock_ has quit IRC17:21
*** prad has quit IRC17:21
*** wenlock has joined #openstack-infra17:21
*** thuc has quit IRC17:22
*** rhsu has joined #openstack-infra17:23
*** dstufft has quit IRC17:23
*** marktraceur has quit IRC17:23
*** mihgen has joined #openstack-infra17:24
clarkbjeblair: you didn't happen to test HPCloud US East yet did you? I haven't heard if they changed perms for us yet17:24
*** dstufft has joined #openstack-infra17:24
*** marktraceur has joined #openstack-infra17:24
jeblairclarkb: glance? no.17:24
*** prad has joined #openstack-infra17:24
openstackgerritKhai Do proposed a change to openstack-infra/publications: Modifications for 2014 Gerrit User conference
*** melwitt has quit IRC17:26
pleia2fungi: if you have time today, I still need the info (cron job, config.js and anything else that looks customized)) from old-wiki for bugdaystats, WIP patch that lacks the info here:
*** melwitt has joined #openstack-infra17:27
fungipleia2: i'll dig that up17:27
*** Ryan_Lane has joined #openstack-infra17:28
*** ildikov_ has quit IRC17:29
openstackgerritA change was merged to openstack-infra/zuul: Allow merge failures to have unique reporters.
*** mkoderer has quit IRC17:31
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Restart unbound with systemd if upstart failed
*** khyati has joined #openstack-infra17:31
derekhthat ^^ should fix the error tripleo-f20 template build is currently hitting, I don't know if there are other problems17:32
derekhbut the rest of that script looks ok17:32
*** leakypipes is now known as jaypipes17:33
*** flashgordon is now known as jogo17:34
*** prad has quit IRC17:34
fungiderekh: any reason not to just look for systemd-managed unbound there rather than assuming any initscript failure should be followed by a call to systemctl? i suspect what you have there could lead to some confusing logs if we ever have to dig into an image build failure related to an initscript issue. lefe a comment17:35
fungier, left17:35
derekhfungi: no reason, will fix17:36
openstackgerritJason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions
*** vhoward has joined #openstack-infra17:40
*** _nadya_ has joined #openstack-infra17:40
*** thuc has joined #openstack-infra17:43
openstackgerritA change was merged to openstack-infra/config: Replace gate-noop with noop
openstackgerritKhai Do proposed a change to openstack-infra/publications: Modifications for 2014 Gerrit User conference
*** thuc_ has joined #openstack-infra17:44
*** prad has joined #openstack-infra17:45
*** aysyd has joined #openstack-infra17:45
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Add support to restart unbound with systemd
*** reed has quit IRC17:47
*** thuc has quit IRC17:48
*** sabari2 has joined #openstack-infra17:52
*** vhoward has left #openstack-infra17:53
openstackgerritA change was merged to openstack-infra/config: Fix centos image name in region-b
*** derekh has quit IRC17:54
*** alff has joined #openstack-infra17:55
*** CaptTofu has joined #openstack-infra17:57
openstackgerritKhai Do proposed a change to openstack-infra/publications: update slides with multi-master jenkins and nodepool
*** _nadya_ has quit IRC17:59
mordredzaro: nice slides! how was our stuff receieved?17:59
fungizaro: what tag do you want on ?18:01
zaromordred: most of the people there were dedicated to gerrit, not to the entire CI work.  so lots of gerrit admins and developers, not much people working on CI in general.18:01
clarkbjeblair: this is a fun mystery, I can't remember why but am digging through logs to see if I can figure it out18:02
zaromordred: so they really liked how we automated gerrit and integrated our tools with it. but not super excited about zuul and jenkins18:03
mordredzaro: I guess I was just wondering if any of how we're using gerrit made sense to them? did they think we were crazy? both? neither? didn't care? :)18:03
mordredzaro: ah - interesting18:03
mordredso manage-projects is a thing they found interesting18:03
zaroyes, very interested.  scaling git and gerrit were also very interesting to the audience18:03
zaroone of the guys created a UI to automate the creation of projects.18:04
zaroi think most of people liked our solution better because they liked that it was reviewable in gerrit.18:05
*** signed8bit has joined #openstack-infra18:05
*** salv-orlando has quit IRC18:06
zarofungi: how about 2014-gerrit-UC ?18:07
*** alff has quit IRC18:07
fungizaro: 2014-gerrit_uc-overview maybe?18:08
zarofungi: wfm18:08
fungithat seems to be a little more in keeping with our year-conference-shortname standard18:08
anteayashould I be able to view the .pngs in that patch somehow?18:09
clarkbanteaya: git review -d and open the file in your web browser18:09
*** SumitNaiksatam has joined #openstack-infra18:09
*** markmcclain has quit IRC18:09
zaroanteaya: i think you just need to view index.html to make sure no images are missing in the slides.18:10
anteayaclarkb: ah18:10
*** zul has quit IRC18:13
fungiheh... 82559 just clicked... that's the board series number for a popular intel ethernet interface i used for years18:14
*** dprince has joined #openstack-infra18:14
fungithose were some of the most stable 100basetx pci cards ever produced18:15
anteayaI see there are tabs in index.html18:15
*** zul has joined #openstack-infra18:16
fungianteaya: yeah, someone's text editor thinks embedded javascript deserves mixed tabs+spaces indents or something18:16
zaroanteaya: not from me :)18:17
fungino, it might be inherited from slidy218:17
anteayazaro: yes they aren't in your edits, they seem to be there from before18:18
anteayazaro: commented18:18
openstackgerritSolly Ross proposed a change to openstack/requirements: Limit psutil to <2.0.0
fungitab substitution is popular in html/javascript because it potentially reduces the number of bytes transmitted over the wire18:18
*** prad has quit IRC18:20
anteayaI can never remember where the publications are published18:20
anteayaI would like to offer a patch to root/readme.rst so I remember18:20
anteayacan anyone tell me where they are published?18:20
clarkbanteaya: iirc18:21
zarojeblair: could you please take a re-look at ?  i would like to land that so i can test it.18:21
anteayaawesome thanks18:23
*** andreaf2 has quit IRC18:23
rcarrillocruzclarkb: hi, when you have some time, may you please revisit ? i exposed the reasons for coding the ref-updated as a special case, as gerrit won't give a branch info on a tag event...18:23
openstackgerritA change was merged to openstack-infra/nodepool: Set paramiko version > 1.9.0
*** morganfainberg_Z is now known as morganfainberg18:24
clarkbrcarrillocruz: looking18:25
*** vhoward has joined #openstack-infra18:25
*** jpeeler has quit IRC18:26
openstackgerritAnita Kuno proposed a change to openstack-infra/publications: Add the publication url to root/README.rst
mtreinishfungi, jeblair, clarkb: if I wanted to make a new branch on a project do I have to do anything special or just do a git review with the new branch name?18:27
*** mestery has joined #openstack-infra18:27
clarkbmtreinish: you have to do special things, it requires bit fiddling in the UI or a force push which no one should have perms for18:28
*** jgallard has quit IRC18:28
jeblairmtreinish: if it's an openstack project, ask the release management team18:28
clarkbmtreinish: they will want to know the project, branch name, and starting ref18:28
*** prad has joined #openstack-infra18:28
mtreinishjeblair: ok, I think cookiecutter is in openstack-dev18:28
*** mihgen has quit IRC18:29
mtreinishdoes that go through the same process?18:29
*** mestery has quit IRC18:29
fungimtreinish: no, but what branch does it need (and why)?18:29
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added prod convenience invocation
mtreinishI was going to add a specs-repo branch for cookiecutter18:29
mtreinishso everything can have a common starting point18:29
mtreinishif we move forward with it18:30
*** e0ne has joined #openstack-infra18:30
*** Hefeweizen has quit IRC18:30
fungimtreinish: for making new specs repo projects?18:30
fungimtreinish: i though cookiecutter didn't support differentiating by branch?18:30
fungimtreinish: if memory serves, that's why oslo has their own cookiecutter project separate from the main one18:30
mtreinishfungi: well I didn't know that18:30
clarkbrcarrillocruz: I understand now. I think it will just have to work that way. Will review latest patchset shortly18:30
jeblairmtreinish: we should get dhellmann and mordred looped into this conversation18:31
rcarrillocruzk, thx sir18:31
dhellmannfungi: yes, that's right, cookiecutter takes a git repo url, but I don't think it handles branches18:31
dhellmannat least, I didn't find any evidence that it does18:31
dhellmannhaving separate repositories feels more sane than branches anyway18:32
dhellmannalthough subdirs in one repo would be cool18:32
dhellmannbut it doesn't handle that, either, afaict18:32
*** Ajaeger1 has joined #openstack-infra18:32
dhellmannwhen I mentioned these things to audrey, she was enthusiastic about adding them, but more so about having pull requests :-)18:33
*** thedodd has quit IRC18:33
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Reduce number of files send for propose_translation
mtreinishdhellmann: ok good to know, so for right now adding a separate specs-repo template repo would be the only option18:35
dhellmannmtreinish: certainly the path of least resistance18:35
fungizaro: is the same conference? if so it looks like maybe they decided that "Gerrit User Summit" is the official name? thinking of doing 'git tag -s -m "Gerr18:35
fungiit User Summit, 2014" 2014-gerrit_user_summit-overview' in that case18:35
fungi(without the stray newline of course)18:36
clarkbrcarrillocruz: how does that look?18:36
zarofungi: yes, and wfm18:37
*** aysyd has quit IRC18:37
*** _nadya_ has joined #openstack-infra18:39
openstackgerritA change was merged to openstack-infra/config: Add jenkins job to build gerrit plugins
rcarrillocruzsure thing, more robust18:39
rcarrillocruzthanks, i will amend18:39
jeblairzaro: do you know if _david_'s wip plugin is ready?  do you want to take a look at it and see if we can/should use it when we upgrade to 2.8?18:40
*** alff has joined #openstack-infra18:40
clarkbrcarrillocruz: fwiw we don't do versions with /s in them but other people may maybe?18:40
zarojeblair: it's ready but not available on 2.8, only 2.9+18:41
clarkbis the old change screen  still available in 2.9?18:41
zarojeblair: i think it would require a patch from master for it to work on 2.818:42
lifelessrussellb: ah ok, I'll tear down this backup copy then :)18:42
zaroclarkb: i'm not sure, don't think so, but will double check.18:42
lifelessihrachys: hi ?18:42
fungijeblair: weird... i get "fatal: reference is not a tree: <sha of tag>" from the publication jobs when pushing a tag for a not-approved change's gitsha. ideas where i went wrong there? the job parameters were
*** thuc_ has quit IRC18:45
*** thuc has joined #openstack-infra18:45
zaroclarkb: yep, old change screen is still availabe:
*** thuc_ has joined #openstack-infra18:46
*** thuc_ has quit IRC18:47
*** thuc_ has joined #openstack-infra18:47
zaroactually if you look thru the slides the new change screen has some very useful features that i didn't even notice when using upstream gerrit.18:47
*** thuc_ has quit IRC18:48
clarkbzaro: yes it has a lot of useful features but a lot of terrible ones too18:48
*** thuc_ has joined #openstack-infra18:49
*** thuc has quit IRC18:49
fungijeblair: looking at the log, i think maybe ggp didn't see the new tag when it did remote update18:50
jeblairfungi: ah, yeah, we should probably see about having ggp update from the zuul merger18:51
*** aysyd has joined #openstack-infra18:51
clarkbjeblair: fungi: oh hey18:51
fungireplicated to the git servers with what looks correct to me...
jeblairfungi: since i believe it will have what it needs...18:51
jeblairfungi: it's probably a race18:51
clarkbabout that, I wholeheartedly agree in making the post queue stuff in g-g-p less racy18:51
*** prad has left #openstack-infra18:51
fungii'll stab at it mightily18:52
clarkbjeblair: fungi also what do you think about hosting /p/ as a valid root path on the replicas to replicate that path on review.o.o?18:52
jeblairfungi: git fetch --tags refs/tags/2014-gerrit_user_summit-overview18:53
jeblairfungi: that got the tag for me18:53
fungijeblair: perfect--i'll have it use that syntax18:53
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add gate jobs for oslo libraries
jeblairfungi: probably best to do that after a git remote update, because it might not have _all_ the tags18:54
jeblairclarkb: why /p/?18:54
clarkbjeblair: because review.o.o and apparently the zuul mergers both support it18:54
clarkbwill make replacing the source of uri paths simpler if all sources support the same paths18:55
*** mrodden has joined #openstack-infra18:55
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Make the oslo.test cross-testing jobs voting
morganfainbergpleia2, fyi i've been swamped.18:57
*** afazekas_ has quit IRC18:57
pleia2morganfainberg: hah, no worries :)18:57
morganfainbergpleia2, might need to wait for the RC to ship before i can think about monitoring collaboration stuffs18:57
morganfainbergpleia2, just letting you know :)18:57
*** mrodden1 has joined #openstack-infra18:57
pleia2morganfainberg: that's fine18:57
anteayafungi: at this point my understanding of salt reactors (something that reacts to an event) is that once an event publishs a tag & data combo, this triggers a salt runner which is a module that executes a command on the salt master, the command can be anything you write and can poll minions for state information or what have you18:58
pleia2thanks for letting me know18:58
*** zns has quit IRC18:58
openstackgerritArx Cruz proposed a change to openstack-infra/devstack-gate: Correcting the right path to mysql log in Fedora 19
anteayathe part I am wondering is that the issue manage projects is facing is that the repo in question needs to be populated across the git farm before manage projects can be executed safely, is that correct?18:59
morganfainbergpleia2, didn't want you to think I wasn't interested still18:59
*** rhsu has quit IRC18:59
anteayaand if yes, how can we get this to send out a tag & data combo to trigger the salt runner18:59
rcarrillocruzclarkb: right, i took to code the stuff. But I rather look at a better understanding to cover more use cases...19:00
fungianteaya: that was a big question for me, at any rate... how to make sure that 'create-cgitrepos' ran successfully on all five git farm servers before running manage-projects on the gerrit server19:00
anteayaI suppose we could poll, but having a completed event say it is completed is tidier19:00
*** mrodden has quit IRC19:00
anteayaah so we both sit here with the same question19:00
anteayaokay will continue to read/dig19:00
*** harlowja has quit IRC19:01
fungianteaya: i had tried to get an example first of using a job worker to trigger git updates on the puppet master and then fire puppet apply on all puppeted servers once that completes successfully, since i thought that might be an easier one to do as a first example19:01
*** rhsu has joined #openstack-infra19:02
anteayadid you have code for that?19:02
*** mrda_away is now known as mrda19:02
fungiUtahDave said he'd put together an example config for that, but doesn't seem to have had free time to work on it19:02
anteayayeah, I keep trying to get 5 minutes of his time19:02
anteayaI'll keep trying19:03
anteayamy sense is that since we won't be moving away from puppet to salt that the powers that be have less free time for Dave to help us19:03
anteayanot unexpected19:03
*** harlowja has joined #openstack-infra19:03
anteayaand he his happy to help when he can19:03
anteayajust less19:03
*** vhoward has left #openstack-infra19:04
*** alff has quit IRC19:04
anteayasince I was asked that question point blank at saltconf and had to tell them no, we won't be moving off puppet19:04
anteayasince we can't19:04
*** SumitNaiksatam has quit IRC19:05
*** _nadya_ has quit IRC19:06
openstackgerritDoug Hellmann proposed a change to openstack-infra/reviewstats: Add requirements project config file
*** melwitt has quit IRC19:09
*** melwitt has joined #openstack-infra19:09
clarkbare the tenant ids and other ids in keystone md5 sums?19:11
clarkbI am taking a stab at making crm114 smarter about those things19:11
*** ociuhandu has quit IRC19:13
*** tianst20 has quit IRC19:14
*** CaptTofu has quit IRC19:14
*** CaptTofu has joined #openstack-infra19:15
*** yonglihe_ has joined #openstack-infra19:15
*** tianst20 has joined #openstack-infra19:15
*** yongli has quit IRC19:16
*** johnthetubaguy1 has quit IRC19:16
fungiclarkb: i believe they vary, diablo era were long int maybe?19:16
fungioh, but at present19:17
fungiand for branches we still care about19:17
*** alff has joined #openstack-infra19:17
anteayaclarkb: I thought they are uuids now19:17
fungii think they're a uuid, just commonly expressed without any punctuation/spacing19:17
bknudsonclarkb: uuid.uuid4().hex19:17
anteayaand were pkis before19:17
openstackgerritClark Boylan proposed a change to openstack-infra/config: Don't treat IDs as uniquely special in CRM114
fungiokay, glad to know i'm not imagining things ;)19:18
mgagnezaro: ping19:18
clarkbbknudson: oh they are rendered that way19:18
clarkbbknudson: I was curious because they don't have the typical -'s separating the chunks19:18
*** UtahDave has joined #openstack-infra19:18
anteayaUtahDave: we were just saying nice things about you19:18
anteayalike how much I missed you19:18
clarkbjeblair: is not tested yet. How were you testing your crm114 changes? just redirecting a local file into crm114?19:19
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Comments UI
clarkbsdague: if you have an example of a log somewhere with PKI token things I can try matching those too19:19
clarkbsdague: so far the two things in 82581 seem to be pretty ubiquitous after scanning a few log files19:19
*** CaptTofu has quit IRC19:19
*** che-arne has quit IRC19:20
sdaguethough I'm not sure we leak PKI into non debug any more19:21
clarkbsdague: it would make me so happy if we didn't :)19:22
clarkbsdague: but I suppose it could be leaked by any change so we should try to deal with it19:22
clarkbfirefox is hating me right now19:22
sdaguewell, it hates these logs in general19:22
*** jpich has quit IRC19:23
sdaguehonestly, I can't interact with the logs at debug level live in anything other than chrome/chromium19:23
*** thedodd has joined #openstack-infra19:25
*** e0ne has quit IRC19:25
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Fetch tag refs from zuul in gerrit-git-prep
*** sandywalsh has quit IRC19:27
*** khyati has quit IRC19:29
*** khyati has joined #openstack-infra19:30
*** yolanda has quit IRC19:31
*** SumitNaiksatam has joined #openstack-infra19:33
clarkbsdague: ok PKI token is going to be tricky because you can only match up to 255 chars usin {N,M} in TRE19:34
*** khyati has quit IRC19:34
clarkband doing things like ([asdfb]{255}){5} are very slow19:34
*** jooools has quit IRC19:34
sdagueyeh, it's probably not worth it19:34
*** hashar has joined #openstack-infra19:36
*** jnoller has quit IRC19:38
zaromgagne: yo!19:39
mgagnezaro: how hard would it be to implement it in JJB until it gets merged?
zaromgagne: maybe connect with a different library?19:41
zaromgagne: i think jenkinsapi already has it that functionality19:42
mgagnezaro: would there be a way to patch it until it's supported?19:42
mgagnezaro: I fear backward compatibility breakages landing in master:
mgagnezaro: I still use 1.x19:43
zarojeblair: ^19:44
mgagnezaro: which triggers a more general question: how far and for how long do we want to support older plugins?19:45
zaromgagne: it's definately possible to do, but i think would require a 2nd connection to jenkins.  jjb uses connection provided by python-jenkins.19:45
* mgagne puts on cowboy hat19:46
mgagnezaro: what about monkey patching?19:46
zaromgagne: that's a good question, my comment on that change was the same question.  would be good to get input from jeblair and clarbk19:47
*** mestery has joined #openstack-infra19:47
mgagneclarkb *19:47
zaromgagne: what monkeys are you thinking?19:48
mgagnezaro: could you inject the method from JJB?19:49
*** zns has joined #openstack-infra19:49
*** dcramer_ has quit IRC19:51
zaromgagne: yeah, i think that will work. but since jeblair can merge stuff to upstream project i think it would be good to get his feedback.19:53
openstackgerritSergey Lukjanov proposed a change to openstack-infra/devstack-gate: Don't enable savanna service for sahara
mgagnezaro: oh, I didn't know that19:55
*** emagana has joined #openstack-infra20:00
zaromgagne: i have asked him to look and merge but i think he's been busy.  it might help if you ask as well.20:00
*** sabari2 has quit IRC20:00
mgagnezaro: no rush =)20:00
*** JoshNang has quit IRC20:00
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Stop logging obsolete channel #savanna
*** sabari2 has joined #openstack-infra20:01
*** imcsk8_lap has joined #openstack-infra20:02
openstackgerritSergey Lukjanov proposed a change to openstack/requirements: Bump min hacking version to 0.8.1
jeblairclarkb: yes20:08
jeblairclarkb: crm11420:08
*** ildikov_ has joined #openstack-infra20:09
*** imcsk8_lap has quit IRC20:10
*** prad has joined #openstack-infra20:11
*** hashar has quit IRC20:13
*** hashar has joined #openstack-infra20:14
*** jp_at_hp has quit IRC20:15
*** SumitNaiksatam has quit IRC20:17
*** prad has quit IRC20:18
* SergeyLukjanov fyi, it's possible now just to upload picture for organisations/user profiles at github -
SergeyLukjanovw/o using gravatar20:21
SergeyLukjanovfungi, ^^20:21
jeblairwow.  that was years in the making.20:21
*** prad has joined #openstack-infra20:21
fungiwhen i do an image search for stackforge logo, this is one of the top hits ;)
portanteSergeyLukjanov: thanks, updated20:23
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/config: Update Murano gate job config
*** harlowja_ has joined #openstack-infra20:27
*** prad has quit IRC20:28
*** jeckersb is now known as jeckersb_gone20:28
*** dcramer_ has joined #openstack-infra20:31
*** mwagner_lap has quit IRC20:31
*** khyati has joined #openstack-infra20:31
*** khyati has quit IRC20:31
*** harlowja has quit IRC20:31
*** prad has joined #openstack-infra20:31
*** marun has quit IRC20:32
zarois there anyway to manually trigger a periodic job?20:32
*** rlandy has joined #openstack-infra20:34
*** imcsk8 has joined #openstack-infra20:34
*** khyati has joined #openstack-infra20:35
*** dcramer_ has quit IRC20:36
fungizaro: zaro a zuul admin can run on the zuul master server, though no idea whether that has all the necessary parameters for periodic pipeline20:36
*** marun has joined #openstack-infra20:40
sdaguejogo: yeh, that should shoot the conductor logs down a bunch20:40
sdaguethere are some other goofy oslo log defaults that should be dealt with as well20:40
*** rediskin has joined #openstack-infra20:42
zarofungi: so would it be difficult to run this periodic job?
fungizaro: periodic-gerrit-plugin-javamelody-gerrit-plugin ?20:44
imcsk8hello, we're having a problem in when packstack tests run on the jenkins jobs, one of the tests is no passing with NetworkError: Unknown hostname which is odd because that address is active. do you know who can help me with this? this is one of the outputs:
zarofungi: yes.20:44
fungizaro: i guess if you know what parameters zuul is likely to pass for it when it runs, i can try to trigger it from the command line with the appropriate options20:46
fungii could probably try to infer relevant parameters from another periodic job which has run recently20:46
fungibut i assume that job hasn't been run before20:46
rediskinhi all. tell me please how differs "check-*" and "gate-*" named jobs?20:46
*** markmcclain has joined #openstack-infra20:47
fungirediskin: some jobs have "check-" and "gate-" variants so that we can run the check versions on lower-performing or experimental systems20:47
*** e0ne has joined #openstack-infra20:47
fungiwithout having them impact jobs running in the gate pipeline20:47
zarofungi: ohh actually.  i can just wait to see if it spits out anything tommorrow i guess.20:48
rediskinfungi: thanks20:48
jeblairimcsk8: '' resolves correctly on centos6 node in hpcloud az220:48
*** dcramer_ has joined #openstack-infra20:48
*** markmcclain1 has joined #openstack-infra20:49
jeblairimcsk8: i can tell you that node is running a local caching resolver that forwards queries to, and that it's the only configured namesever.20:49
jeblairimcsk8: maybe that will help.20:49
*** vkozhukalov has quit IRC20:49
*** markmcclain1 has quit IRC20:49
*** salv-orlando has joined #openstack-infra20:49
zarofungi: do you know how i would be able to get the log for that periodic job?  would i just need to look thru all the gerrit masters for the log?20:49
jeblairimcsk8: (on a side note, a lot of folks, especially packagers, aren't too keen on unit tests that access the network)20:50
fungizaro: it will appear under
*** markmcclain1 has joined #openstack-infra20:50
*** markmcclain1 has quit IRC20:50
zarofungi: ohh cool. thanks.20:50
jogosdague: my take on logs is we should have a log level that is: openstack_debug and debug_all20:50
jogoor something20:50
jogowhere openstack_debug doesn't set debug for all dependencies20:51
jogois that feasable?20:51
*** rossella_s has quit IRC20:51
imcsk8jeblair: yes, that test is not my favorite. i wanted to know if there was a problem on your end before i modify the test20:51
*** markmcclain has quit IRC20:51
sdaguejogo: yeh, this was my debug2 idea20:51
jogosdague: you think its too ale to do for icehuose?20:52
imcsk8jeblair: the weird part is that it just started failing today20:52
*** mrodden1 is now known as mrodden20:53
jeblairimcsk8: not that i can tell, but the addition of the local nameserver is a recent change, so it may be causally related20:53
jeblair(not that i can tell == there is not a problem on the test node, that i can tell)20:53
*** jpeeler has joined #openstack-infra20:53
*** jpeeler has quit IRC20:53
*** jpeeler has joined #openstack-infra20:53
*** jpeeler has quit IRC20:54
*** jpeeler has joined #openstack-infra20:54
*** jpeeler has quit IRC20:54
*** jpeeler has joined #openstack-infra20:54
clarkbsorry back from lunch20:55
*** markmcclain has joined #openstack-infra20:55
imcsk8jeblair: i guess this test wasn't designed with this kind of setup in mind, i'll change it so it works with this type of setup. thanks for the help!20:55
sdaguejogo: no20:55
sdagueI think all that's too disruptive right now20:56
sdagueI think it's juno 1 stuff20:56
clarkbit is too nice outside to have a short lunch20:56
*** thuc_ has quit IRC20:56
jogosdague: in that case  just disabling oslo.messaging logs?20:57
*** luis_ has joined #openstack-infra20:57
*** SumitNaiksatam has joined #openstack-infra20:58
sdaguejogo: and lockutils while you are at it20:58
jogoso there may be another way actually20:58
*** luisg has quit IRC20:59
*** sabari2 is now known as sabari20:59
jogolooking into it now20:59
sdaguejogo: there are a lot of ways to skin this cat21:00
*** jeckersb_gone is now known as jeckersb21:01
jogoif this works it solves the problem of needing the oslo sync every time21:01
sdaguejogo: well the issue there is it requires logging.conf installation to get the right behavior21:01
sdaguethe defaults end up being wrong21:01
sdaguethe real issue is we have to get the logging default levels out of oslo code21:01
sdagueand able to set that in nova code21:02
jogosdague: we can fix logging.conf21:02
sdaguethat's bothered me a lot21:02
jogoand this wouldn't solve the oslo part but work around it for now21:02
sdaguejogo: sure, but that requires people install that21:02
sdaguewhich we've never required before21:02
jogosdague: yeah ... true21:02
jogoit would fix gate but no one else21:02
dhellmannsdague: is planned for juno, so if you could write up your thoughts on that we can take them into account21:02
sdagueand logging.conf is powerful, but it's super complicated21:02
*** Sukhdev has joined #openstack-infra21:03
sdagueespecially as we don't use well understood names in openstack components21:03
dhellmannsdague: even a quick email to me or the ml would do it, I can stick them into the release requirements for that lib21:03
sdaguejust randomly whatever module you happen to be in21:03
sdaguedhellmann: sure21:03
jogoif we update the logging.conf sample and better document it, I was thinking this may be a good interam solution for icehouse21:04
*** rlandy has quit IRC21:06
*** bogdando has quit IRC21:06
jogosdague: we don't use even copy the file in devstack ... so not as quick a solution as I thought21:06
*** afazekas has joined #openstack-infra21:07
*** bogdando has joined #openstack-infra21:08
*** pdmars has quit IRC21:09
*** thuc has joined #openstack-infra21:09
*** Ajaeger1 has quit IRC21:09
*** thuc has quit IRC21:10
jogodhellmann: thoughts on putting in oslo-incubator and cherry picking it out to nova etc21:10
jogofor icehouse21:10
dhellmannjogo: looking21:10
*** thuc has joined #openstack-infra21:10
jogoit drops file size dramatically21:10
jogoand an operator can turn this back on via logging.conf21:11
*** luis_ has quit IRC21:11
sdaguejogo: can you do the lockutils one as well :)21:11
jogosdague: yup21:11
sdaguelockutils debug is pointless21:11
*** thuc has quit IRC21:11
dhellmannjogo: that looks like a reasonable change21:11
*** thuc has joined #openstack-infra21:11
dhellmannjogo: you could also just say "oslo = INFO"21:11
jogodhellmann: ohh I like that21:11
* dhellmann is writing up plans for 9 new oslo libraries next cycle21:12
*** thuc has quit IRC21:12
jogosdague: lockutils is incubator right?21:12
jogoif yes, how do we turn it off?21:13
*** thuc has joined #openstack-infra21:13
sdaguejogo: yes21:13
sdagueoh, right, grrr21:13
sdaguethat's the whole oslo sync problem21:14
jogosdague: heh yeah21:14
sdaguebecause it renames things21:14
*** thuc_ has joined #openstack-infra21:14
sdaguedhellmann: is there a sane way to override this from nova code?21:14
*** ociuhandu has joined #openstack-infra21:14
dhellmannjogo: if you put oslo.openstack.common.lockutils in that config option default, I think the sync code will replace "oslo" with "$project"21:14
dhellmannif that doesn't work, you'd have to call log.set_defaults()21:15
jogodhellmann: would that break a sync, if I add oslo=INFO into oslo-sync?21:15
dhellmannjogo: heh, yeah :-/21:15
sdaguejogo: also n-cpu dropped by half there21:15
*** andreaf has joined #openstack-infra21:15
jogodhellmann: lolz21:16
dhellmannset_defaults() doesn't override this value, but I think we're doing it elsewhere21:16
dhellmannjogo: the problems we make for ourselves, right?21:17
jogodhellmann:  oh yeah21:17
fungianteaya: do you happen to know whether came up for a vote in the neutron weekly meeting or something?21:17
*** thuc has quit IRC21:18
jogodhellmann: heh yup just tried an and it broke things21:18
jogoI now have nova=INFO21:18
dhellmannjogo: yeah, I should have seen that one coming21:19
jogoso whats the right step forward for icehouse?21:19
jogosdague dhellmann:21:19
sdaguejogo: oslo.openstack=INFO ?21:20
sdaguejogo: oslo.openstack=INFO, won't that translate well?21:21
jogothat would work for lockutils but not oslo.messaging21:21
sdagueright, do them both21:21
sdagueor, oslo -> nova ?21:21
sdagueso we get nova.messaging ?21:21
jogowe can't add oslo=INFO because, sync turns that into nova21:21
clarkbwow crm114 really chews through CPU :)21:21
* jogo hides while sdague rages 21:21
clarkbjeblair: I am just about done testing my crm114 change21:21
sdaguehonestly, fork log.py21:21
dhellmannjogo: there's a special dispensation for oslo.config in (line 158) so we could add oslo.messaging handling there, then you could add a default for oslo.messaging and it should work21:22
sdagueor that ^^^21:22
sdagueI am going to try not to rage quit over oslo sync21:22
* dhellmann adds another step to
sdagueI'll save that for juno21:22
*** e0ne has quit IRC21:23
dhellmannsdague: I am, seriously, doing everything I can to move us to as many libraries as possible during juno. promise.21:23
sdaguedhellmann: I know, it's not your fault21:23
sdaguewe have a bunch of debt to dig out of21:23
*** dkranz has quit IRC21:23
sdaguejust this config issue has driven me nuts for a while21:24
dhellmannsdague: yep, I'm just trying to cheer you up :-)21:24
*** dizquierdo has joined #openstack-infra21:24
clarkbjeblair: yup seems to work21:24
dhellmannfwiw, I'm going to need some input from the qa and infra teams about how to deal with releases of those libs during the cycle -- alphas, published where, etc.21:24
*** alff has quit IRC21:25
jogodhellmann: that worked21:25
dhellmannjogo: \o/21:25
sdagueclarkb: so on crm114, this is a reason that I was suggesting just cut the line until the ] and be no more clever than that21:25
clarkbsdague: jogo jeblair is ready to review21:25
clarkbsdague: right but how do you make the rule specific enough21:25
*** sandywalsh has joined #openstack-infra21:25
clarkbsdague: cut until the ] is too generic21:26
clarkbsdague: and there are UUIDs everywhere21:26
clarkbsdague: routes.middleware [-] Match dict: {'action': u'show', 'controller': <nova.api.openstack.wsgi.Resource object at 0x2e4ac10>, 'project_id': u'e6372917a0b3465eb8a85b9733e656c5' for example21:26
sdaguethat's at INFO?21:26
*** dprince has quit IRC21:26
clarkbsdague: that particular example was a debug line /me filters down and looks for more examples21:27
sdagueyeh, this was mostly about INFO21:27
sdagueyou are right, there are some UUIDs for sure there21:27
clarkbsdague: nova.osapi_compute.wsgi.server [req-d88d7c64-d8b0-4bad-9663-cc9258946b58 FlavorsExtraSpecsNegativeTestXML-185396789 FlavorsExtraSpecsNegativeTestXML-1496176899] "PUT /v2/7d77450c960a46329c0c63d1ddc55244/flavors/1803403985/os-extra_specs/key1 HTTP/1.1 in the url21:27
sdaguerequest urls for sure21:27
clarkbso this way we catch all of them and replace them21:27
jogodhellmann sdague:
sdaguejogo: beside the fact that it will fail pep8?21:28
jogosdague: fixed'21:28
sdagueoslo.test a released library now as well?21:28
*** rossella_ has joined #openstack-infra21:29
sdagueI'm trying to think of what else might need it21:29
dhellmannsdague: yes21:29
dhellmannsdague: although that's oslotest21:29
dhellmannand nothing in the incubator is using it, yet21:29
sdagueare we going to make it osloconfig and oslomessaging in juno as well?21:29
sdagueand solve our namespace problem21:29
dhellmannsdague: no, because that would mask another issue21:30
dhellmannthe real problem there is we have 2 versions of the library installed21:30
dhellmannwe get a hard error because the namespace package breaks21:30
dhellmannwithout that, we would have weird "why is it using the wrong version of my code" errors21:30
dhellmannmore subtle, and hard to track down21:30
*** fifieldt has quit IRC21:30
*** e0ne has joined #openstack-infra21:30
*** jamielenz is now known as jamielennox21:33
*** HenryG has quit IRC21:33
*** miqui has quit IRC21:33
jeblairclarkb: how do you tell testr to run a single scenario?21:34
clarkbjeblair: test run --parallel testnameregex21:34
clarkbin the case of testscenarios they create new tests with unique names, you should be able to pass a regex to match only the scenarios you want21:35
*** julim has quit IRC21:35
jeblairclarkb: okay, i'm guessing there's something funny going on with regex and the () in the name from scenarios21:36
jeblairi'll throw backslashes at the problem and see if it goes away21:36
clarkbjeblair: ok, let me know how it goes and if you give me an example I can try debugging too (assuming it persists)21:36
*** amcrn has quit IRC21:36
*** dangers is now known as dangers_away21:38
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs
anteayafungi: I will get some info about
anteayafungi: I don't know anything off the top of my head21:39
fungianteaya: okay, thanks. as usual i don't want to be the one making that decision because the outcome doesn't really affect me21:41
jogoclarkb: one more for you
* anteaya nods21:42
*** mfer has quit IRC21:42
sdaguedhellmann: I thought it was venv vs. namespace vs. editable install21:43
sdagueand the venv did the right thing if it wasn't namespace module21:43
dhellmannsdague: we've had cases where they were installed outside of a venv, too21:44
sdaguebecause otherwise we'd hit issues with unit tests all the time21:44
dhellmannmost of the unit tests configs don't allow site-packages, afaik21:44
dhellmannthe *global* site-packages, that is21:44
sdaguedhellmann: nova does21:44
*** jpeeler has left #openstack-infra21:44
sdagueotherwise libvirt tests don't run21:44
dhellmannsdague: right, I said most don't21:44
sdagueand this is definitely a rage quit issue for nova folks atm21:45
sdaguebecause it means they actually often can't run unit tests on devstack nodes21:45
sdaguein any way21:45
*** mriedem has quit IRC21:46
*** fifieldt has joined #openstack-infra21:46
dhellmannsdague: if a non-namespace version of a library ends up installed editable globally and from pypi locally, which copy is importable from the venv?21:47
sdaguedude, I have no idea :)21:47
*** nati_ueno has quit IRC21:47
sdaguethis is all so far out of my depth21:47
sdaguebut at least it won't be21:47
sdagueERROR: oslo.config not found21:47
dhellmannbecause that's the bigger concern for me: tests that fail in subtle ways, rather than with this hard issue we know how to fix21:47
sdagueand it's like ffs python, it's right there ->21:48
dhellmannright, it will be AttributeError: oslo.config.cfg.CONF has no attribute some_new_thing21:48
*** nati_ueno has joined #openstack-infra21:48
sdaguedhellmann: so if you have a reproducable way to get us out of this instead21:48
sdaguethat's goodness21:48
dhellmannsdague: which global libraries does nova need? just libvirt?21:49
sdagueright now, I think that's the killer one21:49
dhellmannok, well, there are other ways to manipulate the import path to add that one package to it21:49
dhellmannperhaps we can update nova's tox.ini to do that, instead of allowing all globally-installed packages21:49
*** thomasem has quit IRC21:50
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs
*** alexpilotti has joined #openstack-infra21:54
*** mestery has quit IRC21:55
*** dims has quit IRC21:56
sdaguedhellmann: sure, that would make tons of people happy if it worked21:57
sdaguelike I said, any fix would be a win21:57
sdagueI'm just super out of my depth in helping on this one21:57
dhellmannsdague: yeah; and like I said, I'm worried the "don't use namespace packages" fix is just going to lead to other issues that are harder to track down -- I'll think about this more21:58
sdagueyep, no worries21:58
sdagueit was an unrelated rant21:59
dhellmannsdague: it looks like the approach taken in add2virtualenv doesn't work, because the argument has to be the parent dir of the package :-/21:59
dhellmannso, dist-packages, instead of dist-packages/libvirt21:59
*** _ruhe_ has quit IRC21:59
hogepodgeHm. It looks like the gates for our proejct broke because rake now needs ruby >= 1.922:02
*** khyati has quit IRC22:03
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1295876
uvirtbotLaunchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New]
openstackgerritKhai Do proposed a change to openstack-infra/gearman-plugin: change logging level from info to debug
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs
*** mestery has joined #openstack-infra22:04
*** mestery has quit IRC22:06
*** che-arne has joined #openstack-infra22:07
*** salv-orlando has quit IRC22:09
anteayafungi: markmcclain will respond to the radware email one way or another22:12
fungianteaya: great--thanks22:12
*** sandywalsh has quit IRC22:13
*** dims_ has joined #openstack-infra22:13
openstackgerritA change was merged to openstack-infra/reviewstats: Add requirements project config file
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator
jeblairnodepool's first tests!  <sniff>  :)22:16
clarkbit is a grown up project now22:17
*** Sukhdev has quit IRC22:18
fungihogepodge: we use rake to run puppet lint for changes to openstack-infra/config... should we expect it to be failing as well now?22:20
openstackgerritClark Boylan proposed a change to openstack-infra/config: Don't treat IDs as uniquely special in CRM114
clarkbsdague: jeblair jogo ^ I think I am actually happy with that22:20
mgagnefungi: if you use Gemfile and install rake from there, yes22:20
clarkblatest patchset will get all TOKENS on a line, first one would only replace the first22:20
hogepodgefungi that depends. If you’re using < Ruby 1.9 you’ll see failures22:20
fungimgagne: oh, i think we might use ubuntu's packaged rake instead... checking22:20
*** jhesketh_ has joined #openstack-infra22:20
mgagnefungi: I think that you are using system packages only, not Gemfile22:21
jogoclarkb: re we using CRM114 for anything right now22:22
clarkbjogo: yes, all log files are piped through it22:22
clarkbjogo: and we index the results22:22
fungimgagne: yep, looks that way... okay22:22
clarkbjogo: it has a bug where it falls over in soem cases on lines with those UUIDs, theory is the statistics file is keeping track of too much unique data and flattening it like that will make it better22:23
*** markmcclain has quit IRC22:23
clarkbjogo: more accurate and less falling over22:23
jogoahh error_pr22:23
clarkbjhesketh_: ohai22:23
openstackgerritMaru Newby proposed a change to openstack-infra/config: Allow the neutron functional job to vote
*** jhesketh has joined #openstack-infra22:24
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Run all python jobs for nodepool
jogoclarkb: what do I do with something like this:22:24
jogoerror_pr for a stacktrace22:25
jeblairclarkb: (also, uuids defeate the algorithm and make it less effective)22:25
clarkbjogo: you can do searches like error_pr:[-1000.0 TO -10.0]22:27
*** khyati has joined #openstack-infra22:27
clarkbjogo: those are the lines that are identified as being "bad"22:27
jogoohh negative are more likly to be errors?22:28
jeblairclarkb: that seems strange to duplicate the match/alter lines, why can't you just have the inner loop22:28
jeblairclarkb, jogo: also, before we get too far, should we reverse that?22:28
jogojeblair: I think so,22:29
jeblairclarkb, jogo: really the pR that it's an error should be a positive value, so we should either invert the sign, or change the column name to "not_an_error_pr" or equiv22:29
clarkbjeblair: you can if you don't want the <fromnext> I am optimizing, potentially prematurely22:29
jeblairclarkb: oh fromnext, didn't see that22:30
clarkbjeblair: the first match starts from beginning of line, subsequent ones start after the last match22:30
jogoheh: Build step 'Execute shell' marked build as failure22:30
jogohas a pr of -322:30
*** jeckersb is now known as jeckersb_gone22:30
*** maxbit has joined #openstack-infra22:31
anteayafungi: UtahDave is educating me on this etherpad:
anteayado join us22:31
anteayaand anyone else is welcome too22:31
clarkbjeblair: the changes in are making my brain melt. won't that cause all allocations for particular images to go to one provider when we are not busy as grant will be the min value?22:32
*** signed8bit has quit IRC22:33
*** andreaf has quit IRC22:33
*** thedodd has quit IRC22:34
jeblairclarkb: so each time you add a provider to a request, the allocator redistributes the request across all providers22:35
*** prad has quit IRC22:35
jeblairclarkb: so if we're requesting 4 nodes from 2 providers, the actuall requests it's working with are 2 nodes from each22:35
clarkbjeblair: the ratio calculation in that diff is used for resource constrained situations?22:36
jeblairclarkb: the bug was that when granting those requests, it was shrinking them even more based on sibling requests which may not have even been an issue22:36
jeblairclarkb: yeah, it _should_ be used when constrained, but was actually used all the time22:36
jeblairclarkb: so we ended up always allocating half of the first request when there were two requests22:37
jeblairclarkb: instead, what we want to do is give the first request _up to_ half of what's available22:37
jeblairclarkb: (so the second request can have up to the second half)22:37
clarkbjeblair: gotcha22:38
jeblairclarkb: and yeah, it's kinda brain melty, thus the unit tests :)  they do test the scenario you were worried about22:39
clarkbjeblair: great, I will keep reading then22:39
*** gyee has quit IRC22:40
*** e0ne has quit IRC22:41
hogepodgePinning in my Gemfile fixed the gates, so definitly ‘our’ problem.22:41
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Run all python jobs for nodepool
*** thuc_ has quit IRC22:42
*** afazekas has quit IRC22:42
*** thuc has joined #openstack-infra22:43
nibalizerjeblair: clarkb want to see why the puppet agent isnt running on puppetdb.o.o
jeblairnibalizer: ack will look22:44
* clarkb puts money on mordred stopping it to debug over weekend22:44
*** zns has quit IRC22:46
openstackgerritGregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README
jeblairnibalizer: my guess is that it was never started22:47
*** thuc has quit IRC22:47
jeblairnibalizer: it _is_ configured to start on boot, but i'm guessing the host has never been rebooted, and no one started it manually22:47
jeblairat least, that's what it looks like from the logs22:48
JayFThe jenkins servers have both self-signed and expired certs (see: Is it valuable to file a bug for this? I mean, self-signed is OK, but self-signed and expired just seems a little strange is all.22:48
jeblairanyone know of a reason it shouldn't be started?22:48
*** CaptTofu has joined #openstack-infra22:48
clarkbjeblair: run puppet agent --test --noop and see what the delta looks like?22:48
openstackgerritGregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README
clarkbJayF: I think we are just using the ubuntu provided snake oil certs22:49
clarkbJayF: for simplicity22:49
JayFclarkb: ah, so it just means that machine is older than a year :)22:49
clarkbno looks like the snake oil certs may only be good for a motnh22:49
JayFLike a failed cert is a failed cert, it was just strange enough to see a self-signed+expired warning that I thought it bore mentioning22:50
jeblairnibalizer, clarkb:
JayFIs there a reason there's not a * cert tossed on those boxes? Just money? Or management?22:50
blkperljeblair: nice :)22:51
clarkbwe don't have a wildcard cert and those machiens were deemed to not need a valid cert22:51
clarkbvery few people actually look at jenkins anymore22:51
nibalizerjeblair: thats on puppetdb.o.o and not ci-puppetmaster.o.o ?22:51
jeblairnibalizer: correct22:51
nibalizerthats fine to be on there22:51
clarkbnibalizer: jeblair: I think the puppet.conf file is the same everywhere and puppetdb is just lagging22:52
clarkbjeblair: should be ok to restart puppet on that node if that was the only diff22:52
openstackgerritGregory Haynes proposed a change to openstack-infra/pypi-mirror: Use openstack git urls instead of github
jeblairclarkb: only relevant one.  started.22:52
*** david-lyle has quit IRC22:52
jogowow in a 24 hour period we run 600 to 900 copies of cehck-temepst-dsvm-full22:52
clarkbjogo: is that all?22:54
*** afazekas has joined #openstack-infra22:54
jogoclarkb: that is only that one job22:55
clarkbjogo: during a busy week (feature freeze time, we run ~25k tests a day half of which are tempest give or take)22:55
clarkbjeblair: reviewed the nodepool tests change.22:55
jogoclarkb: that is only for the one job22:55
jogoso figure that corrilates closely to number of patches that go through check queue that have an inegrated check queue test22:56
clarkboh and its check only22:56
jogoclarkb: yeah was trying to get numbers for tripleo to grok22:56
*** rhsu1 has joined #openstack-infra22:58
*** CaptTofu has quit IRC22:58
*** CaptTofu has joined #openstack-infra22:59
*** salv-orlando has joined #openstack-infra23:00
*** rhsu has quit IRC23:00
*** salv-orlando_ has joined #openstack-infra23:01
funginote to self... stop trying to browse a requirements integration job log on a netbook with only 0.5gib ram23:01
clarkbfungi: gg23:02
*** gokrokve has joined #openstack-infra23:02
anteayaha ha ha23:03
sdaguejogo: 6 - 9 00 sounds right, a little higher than I remember, but there are a bunch of new jobs23:03
sdagueI noticed this morning, devstack is running 16 d-g jobs on every check change23:03
*** CaptTofu has quit IRC23:03
clarkbsdague: yeah d-g and devstack both have quite a few check jobs23:03
sdagueyeh, the 3 ironic and savana ones that are nv are kind of brutal23:04
*** maxbit has quit IRC23:04
*** maxbit has joined #openstack-infra23:04
sdaguethe savana ones should be able to largely go away23:04
jogoI wonder how much infra tripleo is gonna need to keep up with check23:04
anteayafungi: UtahDave gave me a place to begin, so I will start with what he outlined and then follow up with questions23:04
sdagueonce we get past release23:04
*** salv-orlando has quit IRC23:04
*** salv-orlando_ is now known as salv-orlando23:04
fungid-g has the issue of being branchless, so needs to be tested against stable branches of everything too23:04
sdagueright, d-g I understand it more23:05
fungianteaya: sounds good--looked like it might be a good start anyway23:05
anteayaand you are back on the etherpad again23:05
sdagueI just hadn't counted the jobs on devstack until this morning23:05
anteayayay for a good direction23:05
fungianteaya: yeah, i stopped trying to look at job logs from here23:05
anteayaha ha ha23:05
*** maxbit has quit IRC23:05
jogosdague: 600-900 was on'y check-dsvm-tempest-full'23:06
jogoand thats it23:06
*** maxbit has joined #openstack-infra23:06
*** CaptTofu has joined #openstack-infra23:07
*** ArxCruz_ has joined #openstack-infra23:07
*** ArxCruz has quit IRC23:07
*** rcleere has quit IRC23:08
clarkbjeblair: reviewed but not approved (comment explains why)23:08
jeblairclarkb: yeah, i didn't want to change the allocator from under the multinode change (which also changes the allocator)23:09
jeblairi actually spotted the bug while developing multinode, but decided to push it off until after23:10
clarkbjeblair: agreed23:10
clarkbjeblair: fungi now that we sort of gate on check tests, would it be inappropriate to -1 that change because it isn't gating on that test?23:10
clarkbimo if we are running check tests and voting on them then we should gate on them too23:10
jeblairclarkb: yes, and in fact because of the hole in "require check for gate" logic, we _need_ to do that23:11
openstackgerritJamie Lennox proposed a change to openstack-infra/config: Add kite notifications to the barbican channel
sdaguejogo: yeh, check-dsvm-tempest-full is a pretty good proxy for "all jobs"23:11
jeblairclarkb: otherwise we can wedge the gate23:11
*** aysyd has quit IRC23:11
clarkboh right23:12
jogosdague: yeah that was my  thinking although I wonder how many stackforge projects use it23:12
sdaguejogo: it's under a different name23:12
jogosdague: oh even better23:13
sdaguejeblair: the neutron job won't wedge the gate if the neutron functional tests aren't gated on23:13
sdagueit will just wedge neutron23:13
sdagueor am I missing a piece of logic23:13
*** salv-orlando_ has joined #openstack-infra23:14
nibalizerzaro: you're running puppet actively on zuul-dev.o.o ?23:14
nibalizerit should be appearing with its stuff in puppetboard23:14
jeblairsdague: what you say is correct, but for neutron, that's "the gate"23:14
clarkbnibalizer: I don't think it is23:15
clarkbI am probably to blame for that, will look into it now23:15
*** dangers_away is now known as dangers23:15
*** SumitNaiksatam has quit IRC23:15
sdaguejeblair: right, but that's fixable completely by neutron cores23:15
sdagueit's not like they are broken by something else23:15
sdagueI'm actually less comfortable with them putting this in the gate, given that it's depending on unstable tempest interfaces.23:16
sdaguewhich actually makes me think it probably shouldn't vote at all23:16
*** dcramer_ has quit IRC23:16
jeblairsdague: correct, however, i think it is undesirable for them to be able to accidentally break their own gate due to a missing piece of logic in zuul, so i believe that if they are voting on it in check, it should be voting in gate for them23:16
*** salv-orlando has quit IRC23:17
*** salv-orlando_ is now known as salv-orlando23:17
clarkbok I am restarting puppet on zuul-dev, the diff shows the unbound changes and the gate-noop to noop zuul layout change23:18
sdagueso I definitely don't think that something which depends on tempest unstable interfaces should be able to cause resets for everyone else in the gate23:18
clarkbit looked like puppet agent had been stuck since february 25th23:18
sdagueso maybe we just say it has to stay non voting23:18
*** hashar has quit IRC23:19
fungiclarkb: github remotes for vcsrepos in /opt maybe?23:19
fungiclarkb: i've been cleaning those up as i stumble across them23:19
fungi(just rm -fr the repo and let vcsrepo reclone it from our git servers on the next puppet run)23:20
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add debugging metrics to RPC
clarkbfungi: oh thanks, there is indeed a hung git process23:20
*** dizquierdo has quit IRC23:21
*** maxbit has quit IRC23:21
* clarkb reruns puppet23:21
clarkbit recloned using git.o.o as origin23:22
jogosdague clarkb: this turn off debug logs for oslo.messaging is shrinking a lot of logs23:22
clarkbjeblair: zaro I think zuul on that host may need to be hit with a hammer to make the zuul merger stuff work right23:22
sdaguejogo: yeh, I honestly think oslo.messaging was the thing that caused the explosion23:23
sdaguethe dates line up right23:23
*** CaptTofu has quit IRC23:25
*** CaptTofu has joined #openstack-infra23:25
jogoceilo logs are really bad still23:26
jogofull of /opt/stack/new/ceilometer/ceilometer/openstack/common/db/sqlalchemy/ DeprecationWarning: BaseException.message has been deprecated as of Python 2.623:27
jeblairclarkb: i did actually run the tests on and they pass.  :)23:27
clarkbjeblair: huh, I really didn't expect that23:27
*** amcrn has joined #openstack-infra23:27
* clarkb runs them23:27
*** bknudson has quit IRC23:28
jeblairclarkb: what you describe was the case if the class is in, so i moved it to __init__ and it did what i wanted23:28
clarkbjeblair: interesting and good to know thanks23:29
clarkbI will update my review shortly23:29
jeblairclarkb: if that's a bug, don't tell anyone, i think it's good behavior.  :)23:29
*** CaptTofu has quit IRC23:30
clarkbsure enough it doesn't run them out of __init__.py23:30
*** homeless has quit IRC23:35
clarkbzaro: jeblair is that change being made after operational observations?23:35
*** wenlock has quit IRC23:36
clarkbI think zaro is commuting right now so may not be able to answer, but a bit more background would be handy23:36
sdaguejogo: the tempest logs should be about 60% smaller now23:36
jogoI saw you had a flurry  of patches doing things there23:38
*** salv-orlando has quit IRC23:38
*** yamahata has quit IRC23:39
*** afazekas has quit IRC23:40
clarkbjeblair: looks like that behavior is intentional. you need to explicitly load tests in __init__.py23:41
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add expand by default option to status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add a filter box to status.html
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs
*** thuc has joined #openstack-infra23:43
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers
*** atiwari has quit IRC23:46
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client
*** gokrokve has quit IRC23:48
morganfainbergclarkb, removing a bunch of keystone log lines from testing
clarkbmorganfainberg: great thanks (keep in mind I think we are happy to keep verbose logs if they are helpful, but would like to see unneeded verbosity trimmed)23:51
*** mriedem has joined #openstack-infra23:52
clarkbjeblair: fungi mordred heads up I just responeded to the hpcloud dnssec thread23:55
clarkbhopefully we can get a proper answer23:55
fungiclarkb: thanks!23:56
clarkbI also just approved a change to stop recording #savanna, I checked the wiki meeting schedule page and no tuesday 0000 meetings are scheduled23:58

Generated by 2.14.0 by Marius Gedminas - find it at!