Friday, 2014-03-28

*** saschpe has quit IRC00:00
jhesketh__heh, nothing exciting, just a rebase00:00
*** shardy has quit IRC00:02
*** shardy has joined #openstack-infra00:03
*** saschpe has joined #openstack-infra00:05
openstackgerritMaru Newby proposed a change to openstack-infra/config: Fix tempest installation for neutron api job
openstackgerritBenjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin
marun fungi, clarkb: ^^ I keep getting little things wrong :(00:07
marunhow do you normally debug these jobs?  do you run things locally?00:07
maruniterating through gerrit is pretty painful for someone ignorant like me.00:08
*** mrodden has joined #openstack-infra00:08
openstackgerritA change was merged to openstack-infra/storyboard: Added pagination to comments API
jeblairmarun: yeah, easiest way is to do it on your own cloud server.  the devstack-gate repo has some instructions on how to do that in the README00:10
*** dstanek has joined #openstack-infra00:10
marunjeblair: danke00:10
*** lcheng has quit IRC00:11
*** lcheng has joined #openstack-infra00:12
jogodo WIP patches get expired?00:13
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Added pagination directives
clarkbjogo in the past they did but I think fungi fixed that behavior00:14
jogoah sweet00:14
clarkbI have a handful of old WIP changes that are still alive00:14
sdaguewould there be a way to configure that per project?00:15
sdagueI honestly find the 6 month old WIP patches to be really distracting00:15
jogoclarkb: even if they have a -1?00:15
sdaguejogo: -1 stuff expires regardless00:15
jeblairsdague: the idea of wip is you aren't supposed to see it00:15
sdagueI see it all the time00:16
*** SumitNaiksatam has quit IRC00:16
jeblairsdague: it doesn't show up in the important changes view; if you're using another view, you might want to filter it out00:16
jogosdague: no way to hide WIPs00:16
jeblairyes you can hide WIPs!  that's the point!00:17
sdaguejeblair: important changes is completely useless to me00:17
sdaguebecause it's 600+ reviews00:17
*** _nadya_ has joined #openstack-infra00:17
jeblairsdague: great, fine, don't use it, but if you work based on searches, then you should consider filtering things you don't want to see, like wip, from those.00:18
sdagueok, how do I filter wip ?00:18
sdagueI guess I didn't realize I code00:18
maruni think it'd be useful to be able to query by things like bug/blueprint severity…00:18
*** reed has quit IRC00:18
clarkbsdague: -status:workinprogress iirc00:18
sdagueclarkb: nice00:19
sdagueok, that's helpful00:19
clarkbmarun: there is a thing, reviewday00:19
marunclarkb: ooh00:20
jeblairmarun: or consider writing a patch to gerrit that adds a priority field to changes.  then we could set it based on launchpad status.00:20
marunclarkb: that dprince, always doing cool things I don't know about00:20
marunjeblair: how friendly are they do patch submissions?00:20
marundo -> to00:20
marunjeblair: I can think of a bunch of changes to make :)00:21
marun(not that I have any love for java, mind)00:21
pleia2marun: we're friendly00:21
*** bknudson has joined #openstack-infra00:21
jeblairmarun: i'll be honest, we've had very mixed results; starting on the ml or in #gerrit would probably be advisable.00:21
marunjeblair: ok00:21
jogoso silly question00:21
marunpleia2: so, on the topic of gerrit...00:22
pleia2marun: I meant reviewday :)00:22
marunpleia2: ah!00:22
jogo shows my config patch as in the check queue for 22 minutes and none of the jobs are running yet00:22
*** nkinder has joined #openstack-infra00:22
marunpleia2: sorry.  got mixed up.00:22
*** _nadya_ has quit IRC00:22
jogoI thought the build time was generally <20 minutes for a node to become ready00:22
pleia2I think zaro has had most success working with gerrit folks00:22
jeblairjogo: (also your patch to change the calculation hasn't landed yet)00:22
marunpleia2: is there a hosted version of reviewday?00:22
marunpleia2: ah, so that's where that comes from...00:23
jeblairjogo: we're trying to use hpcloud 1.1 more, which has been throwing servers into error state a lot00:23
marunpleia2: I think someone here mentioned that and it fell out of my head.00:23
jeblairjogo: which prompted this change:
marunpleia2: what do you think of allowing sorting by bug/blueprint severity?00:24
jogojeblair:  yeah I need to test that patch first ...00:24
marunpleia2: and how is the score calculated?00:24
pleia2marun: pretty sure it takes bug severity into account00:24
pleia2marun: there's not much to it if you want to have a browse:
*** ogelbukh has quit IRC00:25
marunpleia2: ah, and of course I can just submit a patch through gerrit00:25
pleia2marun: right :)00:25
marunpleia2: yay infra00:25
*** ogelbukh has joined #openstack-infra00:25
*** alexandra has joined #openstack-infra00:26
marunpleia2: awesome, I definitely have some things I'd like to contribute.00:26
marunpleia2: have to run, ttyl!00:26
*** hogepodge has joined #openstack-infra00:27
*** lcheng has quit IRC00:28
pleia2marun: great!00:28
*** lcheng has joined #openstack-infra00:29
jogojeblair:  yeah I need to test that patch first ...00:29
openstackgerritA change was merged to openstack-infra/config: Add gate-nova-specs-docs job
*** marun has quit IRC00:30
*** matsuhashi has joined #openstack-infra00:31
jogohow long until propagates out00:33
*** wenlock has quit IRC00:34
jeblairjogo: should be out by 01:0000:34
jogoahh thanks00:34
*** lcheng has quit IRC00:44
*** lcheng has joined #openstack-infra00:45
*** thuc has quit IRC00:47
*** thuc has joined #openstack-infra00:48
*** thuc has quit IRC00:52
*** derekh has joined #openstack-infra00:57
openstackgerritAnita Kuno proposed a change to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits
*** markwash has quit IRC00:58
*** hogepodge has quit IRC00:59
*** lcheng has quit IRC01:00
*** dguerri has quit IRC01:02
*** gokrokve has joined #openstack-infra01:03
*** lcheng has joined #openstack-infra01:04
*** dguerri has joined #openstack-infra01:05
mordredanteaya: ^^ nice01:05
openstackgerritA change was merged to openstack-infra/nodepool: Stop waiting for resources in ERROR state
mordredjeblair: I finally found someone at HP Cloud who is interested in looking at the information we produce about how the cloud is doing01:06
mordredjeblair: and when I say interested, I mean got really excited01:06
clarkbmordred: \o/01:08
openstackgerritJay Faulkner proposed a change to openstack/requirements: Add ironic-python-agent to projects.txt
*** UtahDave has quit IRC01:10
sdaguemordred: nice01:11
*** SumitNaiksatam has joined #openstack-infra01:14
openstackgerritA change was merged to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits
*** david-lyle has joined #openstack-infra01:16
anteayamordred: thanks01:18
anteayaand yay for someone at hp being really excited about infra info01:18
*** lcheng has quit IRC01:19
*** ProfFalken has quit IRC01:20
*** ProfFalken has joined #openstack-infra01:21
*** HenryG has joined #openstack-infra01:21
*** dcramer_ has joined #openstack-infra01:24
*** amcrn has quit IRC01:26
*** Ryan_Lane1 has quit IRC01:27
*** Sukhdev has quit IRC01:31
*** derekh has quit IRC01:32
*** nosnos has joined #openstack-infra01:37
clarkbhey does anyone know approximately how many people were at the austin design summit?01:38
clarkbannegentle: jogo mordred jeblair vishy ?01:38
*** khyati_ has quit IRC01:41
* fungi pictures it occupying half a conference room, because the other half was being used to store boxes and excess folding tables01:41
clarkbfungi: you are around \o/ do you have a current ATC count for icehouse?01:42
* clarkb is clearly writing a conference proposal01:42
*** alexpilotti has joined #openstack-infra01:42
fungiclarkb: i haven't generated icehouse-cycle-only atc lists (for summit passes we also include anyone who contributed during the havana cycle too)01:43
fungii can get you some rough numbers, just a sec01:43
clarkbstackalytics indicates integrated is 73201:43
*** yamahata has joined #openstack-infra01:43
*** gokrokve has quit IRC01:43
clarkband openstack is 1,16301:43
clarkbI can use those numbers01:44
fungiit's somewhere around that, probably01:44
*** gokrokve has joined #openstack-infra01:44
fungiclarkb: what conference are you proposing to?01:44
clarkbfungi: I can paste my proposal in a bit when it looks written01:45
*** gokrokve_ has joined #openstack-infra01:45
funginever heard of it--clearly they should want you to come talk ;)01:45
fungii need to propose to a few conferences for later in the year, it's just hard to plan pact the horizon of my house move in a few months01:46
funginice... my dyslexia made it "the open source citizens brigade"01:47
fungilooks like a good excuse to visit portland, but that one is actually smack in the middle of when i'm moving. oh well01:48
*** gokrokve has quit IRC01:48
fungiyeah, end of june is when i have to be out of the old place01:49
fungithough if you want to come see north carolina in the heat of october,
fungii need to submit an abstract for that one before they close the cfp. crap01:52
*** alexandra is now known as alex-luncheon01:52
clarkbfungi: when does their cfp end?01:53
fungithe e-mail i got didn't say, neither does their website01:54
fungilast year was the first one they did in nc. a few hiccups but worked out pretty well (they've done a few in atlanta prior)01:56
*** thuc has joined #openstack-infra01:58
* mordred enjoyed all things open01:58
fungii was just sad that frencil didn't have a big enough room so i couldn't squeeze in to see his demo01:59
*** wenlock has joined #openstack-infra02:01
*** thuc has quit IRC02:03
fungiRyan_Lane: fun!
*** lcheng has joined #openstack-infra02:07
*** bhuvan_ has quit IRC02:08
*** yjiang has joined #openstack-infra02:21
*** lcheng has quit IRC02:21
*** lcheng has joined #openstack-infra02:22
*** saschpe has quit IRC02:24
*** bknudson has quit IRC02:24
*** saschpe has joined #openstack-infra02:24
*** jammyjam has quit IRC02:25
*** alex-luncheon is now known as Alexandra02:39
clarkblifeless: is something that infra needs to fix?02:40
uvirtbotLaunchpad bug 1298731 in openstack-ci "ssh command limits for toci virsh don't match ironic commands" [Undecided,New]02:40
clarkblifeless: want to make sure that doesn't get lost somewhere02:40
*** tchaypo has joined #openstack-infra02:43
*** krotscheck has quit IRC02:43
lifelessclarkb: its in an infra codebase (openstack-infra/tripleo-ci)02:44
lifelessclarkb: but it in terms of resourcing02:44
lifelessclarkb: I suspect NobodyCam will pick it up as he's been working on tripleo integration for a bit02:45
clarkbgreat thanks. (just making sure you weren't blocking on us for something)02:45
*** medieval1_ has joined #openstack-infra02:46
*** Sukhdev has joined #openstack-infra02:48
*** mgagne has quit IRC02:49
*** medieval1 has quit IRC02:49
*** zhiyan_ is now known as zhiyan02:50
*** dims has quit IRC02:53
*** jhesketh_ has quit IRC02:56
*** paul-- has joined #openstack-infra03:04
*** matsuhashi has quit IRC03:04
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id
openstackgerritA change was merged to openstack-infra/zuul: Add demo support for openstack's live zuul feed
*** packet has joined #openstack-infra03:07
*** matsuhashi has joined #openstack-infra03:10
*** lcheng has quit IRC03:10
*** nosnos has quit IRC03:17
jogoclarkb: I heard it wasn't very many at austin <50?03:19
jogoclarkb: but i wasn't there03:19
*** harlowja is now known as harlowja_away03:19
*** changbl has joined #openstack-infra03:20
*** pcrews has quit IRC03:24
*** wenlock_ has joined #openstack-infra03:25
*** mgagne has joined #openstack-infra03:29
*** matsuhashi has quit IRC03:31
*** gokrokve_ has quit IRC03:32
*** dcramer_ has quit IRC03:33
*** jamespage has quit IRC03:35
*** jamespage has joined #openstack-infra03:36
*** jamespage has joined #openstack-infra03:36
jogosilly pypi curl03:36
*** dcramer_ has joined #openstack-infra03:46
*** alexpilotti has quit IRC03:46
*** ProfFalken has quit IRC03:47
*** ProfFalken has joined #openstack-infra03:48
*** jdurgin has quit IRC03:48
*** rcleere has joined #openstack-infra03:53
*** matsuhashi has joined #openstack-infra03:53
*** thuc_ has joined #openstack-infra03:53
*** mgagne1 has joined #openstack-infra03:56
*** Ryan_Lane1 has joined #openstack-infra03:57
*** Ryan_Lane has quit IRC03:57
*** Ryan_Lane1 is now known as Ryan_Lane03:57
*** Ryan_Lane has joined #openstack-infra03:57
*** mgagne has quit IRC03:59
*** jdurgin has joined #openstack-infra04:01
*** lcheng has joined #openstack-infra04:02
*** nosnos has joined #openstack-infra04:06
*** mgagne1 has quit IRC04:07
*** mgagne has joined #openstack-infra04:08
*** vkozhukalov has joined #openstack-infra04:15
*** chandankumar_ has joined #openstack-infra04:23
*** chandankumar_ has quit IRC04:23
*** wenlock_ has quit IRC04:25
*** lcheng has quit IRC04:26
*** wchrisj has quit IRC04:27
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev
*** wenlock has quit IRC04:33
*** rcleere has quit IRC04:34
*** dstanek has quit IRC04:40
*** yjiang has quit IRC04:40
*** packet has quit IRC04:43
*** Sukhdev has quit IRC04:48
*** Shrews has quit IRC04:51
*** Shrews has joined #openstack-infra04:52
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: set default configurations for non-test mode
*** nicedice has quit IRC05:01
*** masayukig has quit IRC05:02
*** jhesketh_ has joined #openstack-infra05:02
*** dstanek has joined #openstack-infra05:09
*** nati_ueno has quit IRC05:09
*** wchrisj has joined #openstack-infra05:11
*** tchaypo has quit IRC05:12
*** alff has joined #openstack-infra05:13
*** alff_ has joined #openstack-infra05:13
openstackgerritA change was merged to openstack-infra/zuul: Add in progress bars for jobs
openstackgerritA change was merged to openstack-infra/zuul: Format change headers like infra's status page
openstackgerritA change was merged to openstack-infra/zuul: Add a total progress bar to each change
*** paul-- has quit IRC05:26
*** marun has joined #openstack-infra05:27
*** dstanek has quit IRC05:27
*** wchrisj has quit IRC05:28
*** gyee has quit IRC05:33
*** alff__ has joined #openstack-infra05:34
*** alff has quit IRC05:36
*** markwash has joined #openstack-infra05:39
*** afazekas has joined #openstack-infra05:41
*** tchaypo has joined #openstack-infra05:45
*** afazekas has quit IRC05:46
*** skraynev_afk is now known as skraynev05:48
*** mrmartin has joined #openstack-infra05:48
*** alff_ has quit IRC05:49
*** alff__ has quit IRC05:49
*** maxbit has quit IRC05:51
*** talluri has joined #openstack-infra05:59
*** ArxCruz has quit IRC06:00
*** afazekas has joined #openstack-infra06:01
*** thuc_ has quit IRC06:03
*** thuc has joined #openstack-infra06:03
*** mrda is now known as mrda_weekend06:06
*** _nadya_ has joined #openstack-infra06:07
*** thuc has quit IRC06:08
*** tchaypo has quit IRC06:11
*** tchaypo has joined #openstack-infra06:14
*** vkozhukalov has quit IRC06:15
*** vkozhukalov1 has joined #openstack-infra06:15
*** mrmartin has quit IRC06:16
*** _nadya_ has quit IRC06:17
*** _nadya_ has joined #openstack-infra06:19
*** gtt116_ has joined #openstack-infra06:22
*** gtt116 has quit IRC06:22
*** ildikov_ has quit IRC06:27
*** basha has joined #openstack-infra06:30
*** Ryan_Lane has quit IRC06:35
*** saju_m has joined #openstack-infra06:38
*** dstanek has joined #openstack-infra06:39
*** denis_makogon has joined #openstack-infra06:39
openstackgerritA change was merged to openstack-infra/zuul: Add support to status page to collapse changesets
openstackgerritA change was merged to openstack-infra/zuul: Shorten update spinner fadeout on status page
openstackgerritA change was merged to openstack-infra/zuul: Add a filter box to status.html
openstackgerritA change was merged to openstack-infra/zuul: Add expand by default option to status page
*** vkozhukalov1 has quit IRC06:49
nibalizerdoes anyone know where the puppetmaster-passenger package is specifically installed in the openstack-infra/config repo?06:56
nibalizeri don't see it?06:56
nibalizeris there extra setup scripts called on puppet masters?06:56
*** _nadya_ has quit IRC07:03
*** dstanek has quit IRC07:04
*** Manishanker has joined #openstack-infra07:06
*** mihgen has joined #openstack-infra07:09
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
*** thuc has joined #openstack-infra07:14
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
*** tianst20 has quit IRC07:18
*** thuc has quit IRC07:19
*** matsuhashi has quit IRC07:20
*** matsuhas_ has joined #openstack-infra07:24
*** katyafervent_awa is now known as katyafervent07:25
*** markwash has quit IRC07:28
*** dstanek has joined #openstack-infra07:32
*** jlibosva has joined #openstack-infra07:33
*** mgagne has quit IRC07:38
*** dstanek has quit IRC07:48
*** markwash has joined #openstack-infra07:49
*** ildikov_ has joined #openstack-infra07:49
*** yjiang|2 has joined #openstack-infra07:50
*** mihgen has quit IRC07:53
*** markwash has quit IRC07:55
*** Alexandra has quit IRC07:57
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id
*** flaper87|afk is now known as flaper8708:05
*** ihrachys is now known as ihrachys|wfh08:06
*** vkozhukalov has joined #openstack-infra08:16
*** talluri has quit IRC08:21
*** hashar has joined #openstack-infra08:26
*** jgallard has joined #openstack-infra08:28
*** jcoufal has joined #openstack-infra08:32
*** mihgen has joined #openstack-infra08:33
*** starmer has joined #openstack-infra08:34
openstackgerritBenjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin
*** dizquierdo has joined #openstack-infra08:45
*** dstanek has joined #openstack-infra08:45
*** vkozhukalov has left #openstack-infra08:49
*** basha has quit IRC08:50
*** talluri has joined #openstack-infra08:52
*** matsuhas_ has quit IRC08:53
*** hashar has quit IRC08:55
*** hashar has joined #openstack-infra08:55
*** matsuhas_ has joined #openstack-infra08:57
*** ildikov_ has quit IRC09:03
*** yolanda has joined #openstack-infra09:05
*** wenlock has joined #openstack-infra09:05
*** saju_m has quit IRC09:05
*** pblaho has joined #openstack-infra09:08
*** derekh has joined #openstack-infra09:09
*** wenlock has quit IRC09:09
*** yjiang|2 has quit IRC09:11
*** e0ne has joined #openstack-infra09:12
*** yassine has joined #openstack-infra09:13
*** johnthetubaguy has joined #openstack-infra09:16
*** johnthetubaguy has quit IRC09:17
*** e0ne has quit IRC09:22
*** starmer has quit IRC09:22
*** saju_m has joined #openstack-infra09:25
*** jcoufal has quit IRC09:25
*** jcoufal has joined #openstack-infra09:26
*** dizquierdo has quit IRC09:27
*** e0ne has joined #openstack-infra09:28
*** e0ne_ has joined #openstack-infra09:28
*** markmc has joined #openstack-infra09:31
*** e0ne has quit IRC09:32
*** ildikov_ has joined #openstack-infra09:33
*** jcoufal has quit IRC09:37
*** jcoufal has joined #openstack-infra09:37
*** johnthetubaguy has joined #openstack-infra09:38
*** denis_makogon has quit IRC09:39
*** jp_at_hp has joined #openstack-infra09:40
*** adarazs has quit IRC09:41
*** rediskin has joined #openstack-infra09:43
*** julienvey has joined #openstack-infra09:56
*** andreaf has quit IRC09:58
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Documented module functions.
*** matsuhas_ has quit IRC10:00
*** dizquierdo has joined #openstack-infra10:05
rediskinhi all. I have a jenkins job, but seems I have no sudo permissions on vm10:12
*** yamahata has quit IRC10:12
rediskinis it depends on node? I using bare-precise and bare-centos610:13
rediskinI should use other node? Or there is other way to get sudo?10:13
rediskinhere is the log
openstackgerritRomain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions.
*** nosnos has quit IRC10:16
*** e0ne_ has quit IRC10:16
*** flaper87 is now known as flaper87|afk10:18
*** alexpilotti has joined #openstack-infra10:20
*** e0ne has joined #openstack-infra10:23
*** andreaf has joined #openstack-infra10:23
*** mfer has joined #openstack-infra10:24
rediskinApparently *-bare nodes does not provides sudo permissions. But why?10:25
rediskinI can't even do any apt-get/yum on such nodes10:25
*** david-lyle has quit IRC10:27
*** david-lyle has joined #openstack-infra10:27
*** ominakov has joined #openstack-infra10:30
*** dpyzhov has left #openstack-infra10:31
*** mburned is now known as mburned_out10:32
rediskinAlso I see some jobs using - revoke-sudo on bare-precise node. But there is no sudo permissions on bare-precise node10:33
*** dstanek has quit IRC10:33
sdaguerediskin: I believe that's all working as designed10:33
*** ProfFalken has quit IRC10:34
sdaguewhat is your test attempting to do?10:34
*** ProfFalken has joined #openstack-infra10:34
sdaguemaybe we can work backwards from there10:34
rediskinmy test attempting to install rally. it requires sudo for apt-get/yum install dependencies10:34
*** jcoufal has quit IRC10:34
sdagueyou may be able to do that on the devstack precise nodes10:35
*** jcoufal has joined #openstack-infra10:35
rediskinsdague: understood. what about centos node?10:36
sdagueI think all the centos nodes are multi use, so that's not allowed10:36
rediskinsdague: may I propose change to enable sudo on bare-* nodes? or it will got -2?10:36
sdaguethe unit test nodes, which centos is one, is not designed for that.10:36
sdagueit will get a -210:37
rediskinso how do I get node for my test?10:37
sdagueyou'll have to wait for one of the infra cores to get in channel. SergeyLukjanov and fungi are usually first up10:38
SergeyLukjanovI'm here, reading scrollback right now10:38
*** ociuhandu has quit IRC10:39
SergeyLukjanovsdague, morning10:41
sdaguemorning SergeyLukjanov10:41
*** flaper87|afk is now known as flaper8710:42
* SergeyLukjanov trying to remember details re sudo10:45
*** CaptTofu has quit IRC10:52
*** CaptTofu has joined #openstack-infra10:52
*** CaptTofu has quit IRC10:56
*** amotoki has joined #openstack-infra10:57
*** dstanek has joined #openstack-infra10:57
*** mattymo has joined #openstack-infra10:59
*** che-arne has joined #openstack-infra10:59
*** jamielennox is now known as jamielennox|away11:15
*** dstanek has quit IRC11:17
*** morganfainberg is now known as morganfainberg_Z11:20
*** dstanek has joined #openstack-infra11:20
*** julim has joined #openstack-infra11:21
*** julim has quit IRC11:22
*** rossella_ has joined #openstack-infra11:31
*** rossella_ has quit IRC11:31
*** julim has joined #openstack-infra11:31
*** Shrews has quit IRC11:33
*** Shrews has joined #openstack-infra11:33
*** dstanek has quit IRC11:46
*** mestery has joined #openstack-infra11:47
*** saju_m has quit IRC11:50
*** dkranz has quit IRC11:54
*** mburned_out is now known as mburned11:57
*** yolanda has quit IRC11:57
*** yolanda has joined #openstack-infra11:59
*** dguerri is now known as _dguerri12:01
*** weshay has quit IRC12:06
*** weshay has joined #openstack-infra12:06
*** david-lyle has quit IRC12:06
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id
SergeyLukjanovjhesketh_ ^^12:09
*** weshay has quit IRC12:10
jhesketh_SergeyLukjanov: cool12:11
jhesketh_SergeyLukjanov: I've replied to patchset 412:12
*** weshay has joined #openstack-infra12:12
jhesketh_SergeyLukjanov: did you want to make the attribute tag only applied when actually shortened?12:13
SergeyLukjanovjhesketh_, >10 chars sounds correct12:13
SergeyLukjanovjhesketh_, but for commit id it's correct to cut and not add '...'12:14
*** dstanek has joined #openstack-infra12:14
*** mattymo has quit IRC12:15
jhesketh_SergeyLukjanov: yep, makes sense12:15
SergeyLukjanovjhesketh_, so, I prefer to cut only commit id12:15
SergeyLukjanovjhesketh_, and display change id as is12:15
*** pdmars has joined #openstack-infra12:15
jhesketh_SergeyLukjanov: Why not cut commit-id's at 6 chars and anything else >10 chars with '...'?12:15
*** pdmars has quit IRC12:16
*** pdmars_ has joined #openstack-infra12:16
SergeyLukjanovjhesketh_, I don't think that we could have in something else than change id and commit id12:17
SergeyLukjanovjhesketh_, change id with length 10 is too big for us :)12:17
*** pdmars_ has quit IRC12:17
SergeyLukjanovjhesketh_, so, probably it's a bit useless to add conditions to cover such cases12:17
*** pdmars has joined #openstack-infra12:17
SergeyLukjanovanyway, I can do it if you think that it's really needed12:18
jhesketh_hmm, interesting point12:18
jhesketh_I'm happy not to block on that :-)12:18
SergeyLukjanovgtg - lunch12:18
openstackgerritA change was merged to openstack-infra/config: Fix tempest installation for neutron api job
jhesketh_sure, I'm nearly off to bed12:19
jhesketh_SergeyLukjanov: it'd be good to see the attr only applied to changes that need it though as it reducs clutter12:19
*** mattymo has joined #openstack-infra12:20
sdagueSergeyLukjanov: instead of truncating in code, what about defining a bounding box on the css so it will just cut it off if it takes up too much space?12:20
jhesketh_sdague: using the attr tag allows the user to expand to get the full id12:20
sdagueah, gotcha12:20
SergeyLukjanovsdague, we have a nice on hover fot abbr12:20
*** pdmars has quit IRC12:22
*** mestery has quit IRC12:22
*** pdmars has joined #openstack-infra12:22
sdaguefirst time looking at the app.js extract. Any idea if there is some thinking to do more of these display bits with templates?12:22
jhesketh_sdague: how so?12:23
*** dcramer_ has quit IRC12:24
sdaguejhesketh_: there was a thread a few months ago about which would let you pull all the html building out of js and be a base template12:25
sdaguethat just has variables12:25
jhesketh_well the scope I'm working in is to modularise the existing javascript so I can pull certain things out easily (for example just a change block)12:26
jhesketh_so it's not something I'm thinking of doing, but others may be12:26
sdagueyeh, I just found adding features was getting increasingly cumbersome with the constant need to do .append12:27
*** maxbit has joined #openstack-infra12:27
sdagueanyway, just curious12:28
jhesketh_yeah, it's not ideal12:29
*** adalbas has joined #openstack-infra12:29
jhesketh_it's getting broken into separate format functions at the very least12:29
*** aysyd has joined #openstack-infra12:29
sdaguesure, I suppose that will make it easier to turn into templates later12:30
jhesketh_indeed... I also wonder if there would be much difference between in defining the elements in a template vs handling the status.json yourself12:31
sdaguejhesketh_: meaning?12:31
jhesketh_well if everything were loops and variables in a template I wonder if you wanted to do something different to the template if you should just parse the json yourselve12:32
jhesketh_ie, there wouldn't be much logic in the javascript left to reuse12:32
jhesketh_I'm not sure how true that is, just thinking aloud12:32
sdaguewell I see the current application as old school php12:32
sdagueall the logic and presentation in one blob12:33
sdaguemaybe other folks are differently, but I think about layout best when I'm actually looking at the declarative markup12:33
sdagueand then logic for variable definition in code12:33
jhesketh_yeah, I agree12:34
*** mwagner_bbl has quit IRC12:35
jhesketh_I think more of it along the lines of building objects (ie the dom)12:35
jhesketh_and I guess if you took those out of javascript you'd only be left with the few UI elements to handle in JS (eg the filter box/expand+collapse functions)12:35
sdagueright, then the javascript would just be the controller, basically12:35
sdagueand the view would be in the templates12:35
jhesketh_but the point I was leading to is that if you then wanted to just display a changeset on a separate page for some reason, you may as well parse the json yourself and load it into your html/template (assuming you needed something different to the original template)12:36
jhesketh_not that I think it's a bad idea, I just wonder on the usefulness12:36
*** dstanek has quit IRC12:36
sdaguejhesketh_: yeh, this wasn't really about the changeset issue12:36
sdaguethis was more general on the whole app12:37
sdagueI agree, if you just want snippets, grab the json12:37
jhesketh_and to clarify, I would be keen to see a templated version12:39
sdague - this is a good instance, where there is really only 1 variable there12:39
sdaguejhesketh_: cool, maybe when I need a break from python I'll see about proposing something :)12:40
*** jp_at_hp has quit IRC12:40
*** dkranz has joined #openstack-infra12:41
*** basha has joined #openstack-infra12:42
*** dstanek has joined #openstack-infra12:45
*** rfolco has joined #openstack-infra12:45
*** hashar has quit IRC12:46
*** chandan_kumar has quit IRC12:51
Shrewsfungi: have there been any reports of unit tests failing because of something eating all memory, engaging the OOM killer? I've been seeing this randomly in ironic tests, and Jenkins has now seen it:
fungiSergeyLukjanov: jhesketh_: a minor point, but if you abbreviate git commit shas, best to do 7 digits since that's the same length most of the git command-line tools also abbreviate to12:51
*** basha has quit IRC12:51
fungiShrews: looking12:51
*** ociuhandu has joined #openstack-infra12:53
sdaguesweet... pypi is pooping on certs -
Shrewsfungi: i didn't start seeing this until i rebuilt my tox venv. makes me think the latest version of something is wonky12:53
*** rlandy has joined #openstack-infra12:54
sdagueShrews: I know neutron was hitting those issues, but that was just because of how their tests were written12:54
fungiShrews: definitely sounds like something is leaky... not sure whether it's just one of ironic's requirements, but seems likely or i think this would be more widespread12:54
anteayaShrews: what version of tox are you using?12:54
fungiShrews: could definitely be explained by a new release of one of the reqs though12:55
sdagueShrews: do your unit tests normally take 40 minutes?12:56
Shrewssdague: no12:56
sdaguenotice the run time from testr start to when it blows up12:56
fungiyeah, maybe something went out of control iterating/looping12:56
sdague2014-03-28 00:52:30.332 | running testr12:56
sdague2014-03-28 01:31:14.428 | running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m discover -t ./ ./ --list12:56
Shrewsfungi: if it isn't widespread, it certainly points to something ironic specific12:56
Shrewsbut it's so RANDOM12:57
sdagueShrews: could be some global sharing12:57
fungiif random.random()>0.99: eat_all_ram()12:58
sdagueso when tests are running in different orders 2 conflict12:58
sdagueShrews: welcome to the wild world of hunting down race conditions :)12:58
Shrewssdague: i've visited that world before. swore i'd never go back  :(12:58
* Shrews blames fungi for all the things not working12:58
* fungi feels so special!12:59
*** thuc has joined #openstack-infra13:00
*** thuc_ has joined #openstack-infra13:00
*** ameade has joined #openstack-infra13:01
*** jeckersb_gone is now known as jeckersb13:01
*** alff has joined #openstack-infra13:01
*** alff_ has joined #openstack-infra13:01
*** dkranz has quit IRC13:02
anteayaShrews: for tracking down races in unit tests: testr run --parallel --concurrency N --until-failure13:03
anteayafor full explanation:
anteayatimestamp 2014-02-20T19:14:2513:03
*** thuc has quit IRC13:04
*** dprince has joined #openstack-infra13:05
Shrewsanteaya: thanks13:06
*** alff has quit IRC13:06
*** talluri has quit IRC13:07
*** alff_ has quit IRC13:07
*** jhesketh__ has quit IRC13:08
ryanpetrelloI’ve noticed a py33 test failure for pecan13:10
ryanpetrellothis familiar to anybody?13:10
ryanpetrello"/usr/local/jenkins/slave_scripts/ line 39: .tox/py33/bin/pip: No such file or directory"13:10
fungi/usr/local/jenkins/slave_scripts/ line 34: tox: command not found13:11
fungisomehow tox ended up not being preinstalled on that slave, or else something caused the jenkins user not to find it13:12
fungii'll check another from the same image13:12
fungimake sure we're not dealing with a botched image build13:13
*** zigo has quit IRC13:13
*** lcostantino has joined #openstack-infra13:14
*** zigo has joined #openstack-infra13:14
*** basha has joined #openstack-infra13:14
fungithat ran in hpcloud region b, where we don't have a ton of slaves building compared to other clouds, so it could go unnoticed there for a while if so13:14
*** CaptTofu has joined #openstack-infra13:15
*** eharney has quit IRC13:15
*** saju_m has joined #openstack-infra13:15
*** basha has quit IRC13:15
fungiyep, found another from the same image in the same state, so i'll see what the image build log had to say about that one13:16
*** mriedem has joined #openstack-infra13:17
openstackgerritMark McLoughlin proposed a change to openstack-infra/config: Upload oslo.messaging final releases to pypi
*** markmcclain has joined #openstack-infra13:19
fungi2014-03-28 02:34:29,629 INFO ssl.CertificateError: hostname '' doesn't match either of '*', '', '*'13:20
fungidstufft: ^ any guess what that's about?13:20
*** dkranz has joined #openstack-infra13:20
sdagueyeh, that's hitting a bunch of tests in the gate13:20
sdaguealso... * does not inspire confidence13:20
dstufftyea I do13:20
dstufftWe switched to a different Fastly certificate last night, apparently one or more nodes didn't get the new cert rollout13:21
fungithat's *so* neat!13:21
dstufftlet me check what the TTL is on the DNS entries13:21
anteayafungi: regarding new projects, I see 3 at,n,z13:22
fungirandom pypi https fails for breakfast13:22
anteayafungi: all look good to go to my eyes, but the neutron-specs one won't go I'm sure due to the specs discussion13:23
*** dizquierdo has quit IRC13:23
fungianyway, i'll delete that image, but also this falls into the class of "puppet failed to do something during the nodepool image build, so why didn't it error out and abort the script?"13:23
*** ArxCruz has joined #openstack-infra13:23
fungipabelanger: ^ i know you were looking into ways we might mitigate that13:23
sdague35 test jobs failed so far because of it13:23
*** markmcclain has quit IRC13:24
dstufftthe TTL is 1 day13:24
sdaguefungi: we got an appropriate bug on this yet, or should I file one?13:25
fungiokay, deleted the broken nodepool image and all ready nodes built from it13:25
fungisdague: i haven't started sniffing the morning's bug pile yet, so not sure13:25
fungichecking now13:26
*** thomasem has joined #openstack-infra13:27
fungisdague: bug 129892913:27
uvirtbotLaunchpad bug 1298929 in openstack-ci "failed download pip from pypi" [Undecided,New]
fungialso, bug 1298827 is related but i can mark it resolved now13:27
uvirtbotLaunchpad bug 1298827 in openstack-ci "tox: command not found on unit-test slave" [Undecided,New]
sdaguefungi: great13:28
fungianteaya: thanks for going through those. i'll hopefully have a look at them in a little while13:29
fungii have the first pass of the junk removal contractor coming momentarily to haul away the stuff we don't want to move but can't effectively sell or give away13:30
fungiso my time is going to be a little divided for the next few hours13:30
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: added bug for new pypi cert issue
*** johnthetubaguy has quit IRC13:32
dstufftsdague: fungi sorry y'all :(13:32
anteayafungi: so you have removed the latest images so we are going back to yesterday's images for nodes?13:33
sdaguedstufft: no worries. openstack ci is just nagios for the internet13:33
anteayafungi: when should those be available?13:33
*** basha has joined #openstack-infra13:33
*** basha has quit IRC13:34
fungianteaya: so far it was just one image for one node type in one provider which seemed affected, and we keep the previous day's image on hand, so it'salready switched to that one now13:34
anteayaokay thanks13:34
*** atiwari has joined #openstack-infra13:35
dstufftfungi: sdague Is there a particular DC that the errors are happening in by any chance?13:35
dstufftor a provider13:35
dstufftanteaya: is that to me? :)13:36
funginot sure about the test failures for pypi downloads, but the image which failed to build because of a pypi download issue was in hpcloud region b13:36
dstufftfungi: do you know where that is located geographically?13:36
anteayadstufft: no, that is the node type and provider13:37
anteayain case anyone else is reading the logs13:37
fungidstufft: east coast of the usa i think, but not sure where (hp people in here may have more of an idea)13:37
fungianteaya: well, the pypi download errors could also be happening elsewhere. i was specifically addressing the location of the broken nodepool image which resulted from the same underlying issue, but was causing a different set of job failures13:38
isviridovfungi anteaya I'm about meetbot in #magentodb channel any ideas when it can be merged?
sdaguedstufft: it's pretty spread out -
sdaguelooks like more hp than rax13:39
sdaguebut there is one rax hit13:39
sdagueregion b more affected than others, but it's not limitted to that13:40
anteayafungi: yes, I was just putting "one image for one node type in one provider" into my own words13:40
*** thuc has joined #openstack-infra13:41
*** basha has joined #openstack-infra13:41
*** krotscheck has joined #openstack-infra13:41
*** alexpilotti has quit IRC13:42
*** jgrimm has joined #openstack-infra13:42
*** prad has joined #openstack-infra13:42
*** krotscheck has left #openstack-infra13:43
anteayamordred: where did the patch go, I was sure you had written it, to get long urls to be shortened in channel?13:43
anteayamordred: looking at your patches, I can't seem to find it13:43
*** thuc_ has quit IRC13:44
sdagueanteaya: I thought it was a proposed idea from a bot in someone else's channel13:44
anteayaisviridov: hi I have no ability to merge that patch but I have upvoted13:44
anteayasdague: hmmm, so not an infra bot?13:44
rediskinSergeyLukjanov: so what do you think about enabling sudo in bare-* images?13:44
*** thuc has quit IRC13:45
isviridovanteaya, thank you. But who has?13:45
*** bknudson has joined #openstack-infra13:45
dhellmannI'd like to make the cross-testing jobs for oslo.test voting. When you have time, can I get reviews for, please?13:45
sdaguethe race is on... will  land before the tempest change that failed on that race get's processed?13:45
rediskinSergeyLukjanov: may I propose this change? or it doomed to -2?13:45
anteayaisviridov: SergeyLukjanov fungi clarkb mordred jeblair13:46
*** YorikSar has joined #openstack-infra13:46
*** dcramer_ has joined #openstack-infra13:46
anteayaand fungi is dealing with a contracter at his house this morning13:46
sorenDoes jjb offer any way to do conditionals? I have a job-template that does a bunch of stuff. A few of the projects that use it also have another job. I want to trigger this other job, but only for these few jobs that have it. Does that make sense?13:47
*** basha has quit IRC13:47
*** zigo has quit IRC13:47
*** zigo has joined #openstack-infra13:47
dstufftI don't know how to kibani, how can I turn into a link that shows the node names?13:48
*** basha has joined #openstack-infra13:48
isviridovSergeyLukjanov, hi could you merge ? Looks like there is only one meeting today13:48
sorenIf I could specify "Run after" like in the Jenkins UI, I could just make the subsequent job have that attribute, but since that's just a different view of the "builds to trigger" that obviously won't do.13:48
anteayadhellmann: I upvoted but since I don't fully understand all the ramifications for oslo.test voting, it may not mean much13:48
*** basha has quit IRC13:49
*** prad has quit IRC13:49
fungirediskin: we normally prevent jobs running on those node types from doing things as root to help ensure that developers can run the same tests on their own machines without compromising/destroying them. there are some patches in progress which could allow specific jobs to have root access on those nodes, however i don't think they've all made it through review yet13:49
Mithrandirsoren: you can tell it to try to trigger non-existing jobs, it'll just not do anything.13:49
openstackgerritA change was merged to openstack-infra/elastic-recheck: added bug for new pypi cert issue
*** markmcclain has joined #openstack-infra13:50
sorenMithrandir: Heh.13:50
sorenMithrandir: That's true.13:50
fungirediskin: normally we perform installation tests into python virtualenvs, but if you specifically need to test system-level installation it may make more sense to use devstack-precise nodes and set up a custom devstack-gate gate_hook function which runs the commands you need to test instead13:50
openstackgerritA change was merged to openstack-infra/reviewstats: Documented module functions.
sorenMithrandir: I guess I could just do that.13:50
sorenMithrandir: Hm... Yeah, that should work. Thanks!13:50
rediskinfungi: thanks13:51
Mithrandirsoren: it'll complain in the UI, but it's not broken per se.13:51
*** johnthetubaguy has joined #openstack-infra13:51
*** zigo has quit IRC13:51
Mithrandirsoren: there are some plugins to do conditional builds and such, not sure if those would be more what you're looking for.13:51
*** amotoki has quit IRC13:51
*** markmcclain1 has joined #openstack-infra13:51
*** markmcclain has quit IRC13:51
*** prad has joined #openstack-infra13:52
SergeyLukjanovisviridov, looks like oslo meeting will be in 5 mins13:53
isviridovSergeyLukjanov, yeap. Probably after meeting13:53
dhellmannSergeyLukjanov: there's no meeting today, we didn't call one -- we just have that slot reserved13:53
sdaguedstufft: ah, I was hoping the view would be embedded13:53
*** mwagner_bbl has joined #openstack-infra13:53
dstufftsdague: no it was, I was just poking arond myself13:54
dstufftto look at a different error to the same problem13:54
sdaguedstufft: so go to the left13:54
dstufft(the error pip gives, I was hoping to see what version of pip it was)13:54
sdagueclick on the little arrow next to build_node (which creates a js popup)13:54
dhellmannanteaya: the point of those jobs is to ensure that changes in oslo.test do not land if it will break libraries using oslo.test13:54
sdaguethen click on Score13:54
sdaguethat will take you to a page which facets the results by that term13:54
*** zigo has joined #openstack-infra13:55
dstufftsdague: cool thanks13:55
sorenMithrandir: I considered that. I could conditionally trigger another build based on "{somevar_that_is_only_set_for_these_few_jobs} != ''" or whatnot, but that seems horrible.13:55
sdagueI suppose we probably want to add the target bit to that last bug as well13:56
SergeyLukjanovdhellmann, ok, thx for info13:56
SergeyLukjanovisviridov, approved13:56
*** markmcclain1 has quit IRC13:56
*** e0ne has quit IRC13:58
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add a second kind of message that can trigger pypi fail
*** jp_at_hp has joined #openstack-infra13:58
anteayadhellmann: makes sense to have them voting then13:58
openstackgerritMark McLoughlin proposed a change to openstack-infra/config: Upload oslo.messaging final releases to pypi
*** zigo has quit IRC13:59
*** zigo has joined #openstack-infra13:59
*** CaptTofu has quit IRC13:59
*** CaptTofu has joined #openstack-infra14:00
*** CaptTofu has quit IRC14:00
*** wenlock has joined #openstack-infra14:00
*** krotscheck has joined #openstack-infra14:01
*** jlibosva has quit IRC14:03
*** paul-- has joined #openstack-infra14:03
*** johnthetubaguy has quit IRC14:03
*** johnthetubaguy has joined #openstack-infra14:04
*** wenlock has quit IRC14:05
*** eharney has joined #openstack-infra14:05
openstackgerritSergey Skripnick proposed a change to openstack-infra/config: Run rally-install jobs on devstack-precise nodes
*** wchrisj has joined #openstack-infra14:06
*** signed8bit has joined #openstack-infra14:07
dstufftsdague: fungi is it possible to get one of the systems that the pypi hit failed on and see what ip address resolves for there?14:08
*** russellb is now known as rustlebee14:08
*** dansmith is now known as damnsmith14:09
*** dkranz has quit IRC14:09
*** signed8bit has quit IRC14:09
*** signed8bit has joined #openstack-infra14:10
*** johnthetubaguy has quit IRC14:10
sdaguedstufft: most of these systems are recycled right away14:10
*** vhoward- has joined #openstack-infra14:10
openstackgerritA change was merged to openstack-infra/config: Add OpenStack Meetbot to #magnetodb channel
*** johnthetubaguy has joined #openstack-infra14:11
*** johnthetubaguy has quit IRC14:11
dstufftyea I figured :/ Fastly ran a check on all their pops and it's returning the correct cert on all of them, so trying to get an IP that it's happening on, and I (of course) can't repro locally14:11
*** dkliban_afk has joined #openstack-infra14:12
*** johnthetubaguy has joined #openstack-infra14:12
sdaguethe last e-r change to detect this14:13
sdaguefailed on the same issue -
dstufftsdague: how hard would it be to run a patched copy of pip? would that be super hard?14:14
sdaguewhere would it come from?14:15
sdagueyou could always propose a devstack change with the right tracing and try to rerun it until fail14:15
dstufftah, can you point me to that repo? :)14:16
*** openstack has joined #openstack-infra14:21
*** dkranz has joined #openstack-infra14:22
*** markmcclain has joined #openstack-infra14:23
*** bnemec is now known as beekneemech14:25
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job
*** alexpilotti has joined #openstack-infra14:27
BobBalldstufft: I'm constantly hitting the pypi issue
openstackgerritA change was merged to openstack-infra/elastic-recheck: add a second kind of message that can trigger pypi fail
BobBallIf you want a host, I can easily get one :)14:29
dstufftBobBall: if you can get me a host that would be awesome14:29
fungidstufft: sdague: we do have the ip addresses of machines which saw those failures (they're printed at the start of the job console logs) if it helps at all14:29
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job
*** wenlock has joined #openstack-infra14:30        86053   IN      CNAME
BobBallit's got "global-ssl" in there now14:30
BobBallnot "c.ssl"14:30
dstufftBobBall: yea, what's the IP address that resolves too though14:30 30     IN      A
*** david-lyle has joined #openstack-infra14:31
dstufftBobBall: and that's from a host where it's failing on?14:32
openstackgerritThomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates
BobBallhmmm - I thought so but I'll double check.14:32
thervefungi,  ^^, tried to follow your advices14:32
*** pcrews has joined #openstack-infra14:32
*** Manishanker has quit IRC14:33
fungiBobBall: the cert i'm seeing served from that includes *.python.org14:33
BobBalldstufft: confirmed, yes14:33
fungiopenssl s_client -connect </dev/null|openssl x509 -text14:34
dstufftthat may be the wrong reason though14:34
*** wenlock has quit IRC14:34
BobBallWant access to the host?14:34
dstufftthat was just a gues son my end :)14:34
dstufftBobBall: that would be great14:34
BobBallChuck me a public key14:34
BobBalldone - jenkins@
NobodyCamgood morning Infra, I have a question about
uvirtbotLaunchpad bug 1298731 in openstack-ci "ssh command limits for toci virsh don't match ironic commands" [Undecided,New]14:35
jeblairfungi, sdague: good morning14:36
fungiNobodyCam: lifeless warned us you'd be coming ;)14:36
fungimorning jeblair14:36
dstufftBobBall: you were able to reproduce with pip? or a different command?14:36
NobodyCammorning fungi14:36
NobodyCamfungi: so lifeless said "you need to setup a restricted ssh key with the ci_commands file, and then get ironic running against it"14:36
sdaguemorning jeblair14:37
BobBalldstufft: pip; see logs at workspace/testing/logs/devstacklog.txt.gz14:37
NobodyCamarethere any doc or examples I can read / look over?14:37
dstufftrm requests-2.2.1.tar.gz && pip install -d . requests is succeeding every time14:38
*** rwsu has quit IRC14:38
fungiNobodyCam: i'm not 100% sure i parse his suggestion/bug report, but i think he may mean per-key command restrictions in the ssh configuration14:38
*** e0ne has joined #openstack-infra14:38
fungiNobodyCam: let me dig up some documentation and see if it joves with what you're trying to make work14:38
fungiby jove14:39
jeblairfungi, sdague: btw, rax has opened a few more tickets saying that jenkins slaves must be compromised because they are doing lookups on ''.  the timing of the blacklisting and our logs don't support that for me.14:39
*** wenlock has joined #openstack-infra14:39
NobodyCam:) awesome fungi TY14:39
jeblairfungi, sdague: my current theory is that their blacklisting agent isn't keeping up with ip addresses moving across accounts.  i'm asking for logs with timing info from them to confirm.14:39
fungijeblair: i suspect that to be the case (and have for a while)14:40
fungijeblair: particularly since it explains why we'd get fresh instances in rax which can't do any lookups through their resolvers14:40
BobBalldstufft: running as jenkins or stack?14:40
sdaguejeblair: yeh, that sounds pretty sound14:40
dstufftBobBall: jenkins user14:40
dstufftBobBall: note: I'm not very familar at all with devstack :)14:41
*** rwsu has joined #openstack-infra14:41
sdaguethe joys of the cloud, keeping up with whack-a-mole things like blacklists14:41
BobBallworks as stack too14:42
*** primeministerp has quit IRC14:42
jeblairfungi, sdague: i think that confirms that in this case our work is insufficient to address Bug 127038214:42
uvirtbotLaunchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed]
sdaguejeblair: yes, definitely if they are doing it at the network level14:43
jeblairso we may want to consider either forwarding to on all slaves, or making all slaves recursive resolvers14:43
*** e0ne has quit IRC14:43
*** primeministerp has joined #openstack-infra14:43
jeblair(either one at this point should be a simple config switch)14:43
*** aglarendil has joined #openstack-infra14:43
sdagueso are they actually letting dns out of their network?14:43
fungijeblair: right, probably the only other potentially effective workaround would be to have a nodepool acceptance test which tried to resolve something14:43
sdaguethat would seem to defeat the purpose of dns blacklisting14:43
dstufftBobBall: hrmpf14:43
fungisdague: the point of blacklisting is apparently that dns-based reflection attacks are overloading their resolvers14:44
sdaguenot to actually isolate bad nodes14:44
fungisdague: and they want to reduce the impact on their resolvers from compromised servers14:44
fungisdague: right. it's almost certainly to keep their resolvers from crumbling under load induced by their own compromised customers14:45
*** salv-orlando_ has joined #openstack-infra14:46
jeblairfungi: yeah, the '' thing points in that direction too (it's apparently used in amplification attacks)14:46
fungiit was already pretty bad back when i still worked for an isp, and my friends who continue to do so say that it's gotten markedly worse in the past couple years14:46
*** salv-orlando has quit IRC14:47
* NobodyCam is happy to not wark at a ISP any more!14:49
fungiNobodyCam: so here's a not bad blog post (the gritty details are in the openssh manpages if you want them)
BobBalldstufft: want me to just try to re-stack to see if it fails in the same way?14:51
*** thuc_ has joined #openstack-infra14:51
dstufftBobBall: sure!14:51
dstufftif you can repro with a restack we can add some debugging to devstack14:51
*** salv-orlando_ has quit IRC14:51
*** jooools has quit IRC14:53
BobBallinterestingly it seems to always happen at the same place... i.e. other pip installs have worked?14:53
BobBallit did indeed fail in the same place14:53
dstufftBobBall: hrm14:53
dstufftcan you point me to where it failed at?14:53
dstufftwhat line of code that is14:53
BobBallthe failure log is at
*** jooools has joined #openstack-infra14:54
BobBallthe specific line is used loads though14:54
dstufftthat is strange o.O14:55
fungiBobBall: when you say fails at the same place, you mean specifically when downloading pytz?14:55
*** zehicle_at_dell has quit IRC14:56
dstufft is a failure from earlier (the first report)14:56
dstufftboth using pip 1.4.114:56
NobodyCamfungi: Ty reading over now14:56
dstufftwonder if this is a pip bug14:56
dstufftbut that doesn't make sense14:56
dstufftbecause it works sometimes14:56
dstufftBobBall: if you restack a third time it fails again?14:57
BobBalltrying now14:57
BobBallit's stuck on reading package list - but might have been my fault :)14:57
dstufftif I recall devstack downloads pip from a particular url14:57
dstufftI might throw together a pip fork that prints the ip address that it was connecting on14:58
*** mgagne has joined #openstack-infra14:58
BobBallthird time it worked...14:58
dstufftof course it did14:58
dstufftbecause problems can't be easy14:59
fungithis is why we can't have nice bugs14:59
BobBallah - failed now14:59
*** salv-orlando has joined #openstack-infra14:59
BobBallfailed on pycrypto15:00
BobBallsame error15:00
dstufftof coruse I can't repro locally15:00
*** hogepodge has joined #openstack-infra15:00
BobBallI've got a meeting now - sorry guys15:00
dstufftmy pip doesn't point to pypi at all15:00
BobBallfeel free to play with it and dstufft if you want to add ppl to the authorisedkeys go ahead15:00
dstufftBobBall: cool thanks15:00
*** dizquierdo has joined #openstack-infra15:01
fungidstufft: maybe an easy way out is to run a tcpdump session on low-verbosity and make note of the last port 443 destination the machine hit before the script bailed?15:01
*** mgagne1 has joined #openstack-infra15:02
*** mgagne has quit IRC15:02
fungiassuming the nondeterminism here follows a particular remote ip address, if those remote addresses are being layer-3/layer-4 load balanced it could be hard to tell from just cursory checks, and could need a lot of volume to correlate15:05
* fungi suspects multi-tier load distribution, with a farm of servers behind a non-ssl-terminating proxy at each cdn pop15:06
*** CaptTofu has joined #openstack-infra15:07
*** reed has joined #openstack-infra15:08
ttxfungi: is there an anonymous way to get a list of gerrit changes with the current version ? Or you have to use SSH or HTML screenscraping ?15:09
ttxthe REST API seems to be >=2.615:09
*** yamahata has joined #openstack-infra15:10
sdaguettx: not until gerrit upgrade15:11
fungittx: those are pretty much your options afaik... the two interfaces are ssh and https, the first is always authenticated and the second won't return raw results for search queries15:11
sdaguettx: what are you trying to generate?15:11
ttxsdague, fungi: ok thx15:11
ttxsdague: writing a tool to automate TC agenda creation / email notification15:11
*** rcleere has joined #openstack-infra15:11
*** e0ne has joined #openstack-infra15:12
sdagueyeh, gerritlib is your friend15:12
ttxso I just retrieve a list of open governance changes (title and link)15:12
fungittx: if you wanted to tweak openstack-infra/openstackwatch it could probably start publishing what you want in its rss feed15:12
ttxsdague: yeah, will do15:12
*** sheeprine has joined #openstack-infra15:12
fungittx: would something like but for the governance repo work for your needs?15:13
fungittx: or do you need a more fine-tuned query than just "all open changes"15:14
ttxfungi: bit more fine-tuned. Will just query by ssh15:14
sheeprineHi, just a quick question, is ipv6 ready ? Because there is an AAAA record in the DNS, the host is reponding but the http service is timing out.15:14
fungisheeprine: i'm connecting to it via ipv6 on the address listed in its aaaa rr15:15
fungisheeprine: tried https?15:15
sheeprinewill give it a try15:15
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add tox env for testing jenkins-job-builder
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job
NobodyCamhey hey fungi dumb neubee question.. where are the ssh keys comming from I'm look ing over TripleO's repo ( and do not see it, thou I have had very little coffee sofar this morning :)15:16
sheeprinefungi: http is redirecting me to https15:17
sheeprinebut https hangs15:17
fungisheeprine: i can telnet to the http port on its ipv6 address and get / returns a redirect to https as it should. establishing an ssl socket to the https port on it gives me page content when i get /15:17
*** rlandy has quit IRC15:17
*** packet has joined #openstack-infra15:18
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Make the oslo.test cross-testing jobs voting
fungiNobodyCam: presumably the machines in question are built by nodepool? if so, the best place to start tracking that down would be from the nodepool prep script for it...
dhellmannfungi: thanks for the review on ^^15:20
openstackgerritRyan Petrello proposed a change to openstack-infra/config: Enable voting for pecan pypy tests.
fungidhellmann: lgtm now15:20
dhellmannfungi: great!15:21
ihrachys|wfhis safe to recheck?15:23
uvirtbotLaunchpad bug 1298929 in openstack-ci "failed download pip from pypi" [Undecided,New]15:23
sdagueihrachys|wfh: yes15:23
ihrachys|wfhsdague: tnx!15:24
*** paul-- has quit IRC15:24
sheeprinefungi: I'm testing it from two different locations, and everything is hanging at the SSL handshake15:24
anteayasheeprine: are you behind a firewall or using a proxy?15:25
sheeprineanteaya: nope, no problem accessing other ipv6 servers15:25
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add a test for subnodes
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support
fungisheeprine: i agree that is very, very odd. i'm currently hitting it over a gre tunnel from my house to hurricane electric where i hop on the 6bone... but the server is in rackspace which also peers with hurricane electric so maybe there's an issue at the border15:26
*** hashar has joined #openstack-infra15:26
* fungi pokes at some traceroutes15:26
dstufftfound the rogue server15:27
fungidstufft: hooray! it's awesome that you're better at debugging fastly's services than they are15:27
sheeprineI'm in france btw, and my two machines are behind a proxad peering15:27
ryanpetrellosdague: is this ready to be reverted yet?
fungi...wait, no, not awesome... what's that other word?15:27
fungiterrible. that's what i meant15:27
ryanpetrellosdague: I created in response to it15:27
dstufftfungi: heh, they missed a machine when checking their consistency15:28
dstufftshould be all fixed now15:28
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add de-duplication to periodic triggers
fungithanks again dstufft, as always!15:28
sdagueryanpetrello: they are still failing at a good clip from what I see15:28
sdague#3 on the list15:29
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add tox env for testing jenkins-job-builder
dstufftfungi: unles sof course there are more machines :)15:29
dstufftgotta pick up my daughter though15:29
dstufftbe back in a bit15:29
ryanpetrellosdague: so it looks like it’s an issue across the board w/ the pypy workers, not just certain projects?15:30
ryanpetrellosdague: if that’s the case, I’ll get abandon my pecan voting: true and just wait until this bug is fixed15:30
*** mrodden has quit IRC15:30
sdagueryanpetrello: yes, I think it's across the board15:31
fungiryanpetrello: i think that bug is still outstanding. Alex_Gaynor was talking to the debian/ubuntu/ppa pacakger for pypy about getting it fixed but i haven't seen a new package yet15:31
ryanpetrellookay, got it15:32
fungilooks like maybe the bug didn't get linked in sdague's commit message as related15:32
fungiryanpetrello: we're tracking it with bug 1290562 (though it's not really an infra bug because it has to be fixed elsewhere)15:34
uvirtbotLaunchpad bug 1290562 in openstack-ci "failing pypy jobs with "option --single-version-externally-managed not recognized"" [High,In progress]
ryanpetrellofungi: sounds good15:34
ryanpetrellofor now, I’ll just abandon my pecan change for voting: true15:34
ryanpetrellonow that I understand what’s actually up15:34
*** afazekas has quit IRC15:34
ryanpetrelloI’ll rely on this to be fixed and pypy to default to `voting: true` again at some point15:34
*** saju_m has quit IRC15:35
*** thuc has joined #openstack-infra15:41
*** packet has quit IRC15:41
*** mrodden has joined #openstack-infra15:43
*** malini has joined #openstack-infra15:43
*** thuc_ has quit IRC15:44
*** thuc has quit IRC15:45
malinihello! The Marconi team needs to run tests against a newer version of Mongo (not currently provided at the gate)  & we are considering 3rd part testing for that.15:48
maliniBut I am not sure what the best practices are , except for what is here
maliniIs this a good use case for 3rd party testing?15:49
*** mgagne1 is now known as mgagne15:50
*** markwash has joined #openstack-infra15:50
*** markmcclain has quit IRC15:51
fungibug 1299057 is why i -1 new project changes which import from remote repos with excess topic branches. several of the 7 projects they want cleaned up have 50+ unwanted branches15:52
uvirtbotLaunchpad bug 1299057 in openstack-ci "Remove outdated branches in fuel-* repos" [Wishlist,In progress]
jeblairfungi: ++15:53
openstackgerritA change was merged to openstack-infra/nodepool: Raise min_demand due to slow node boot times
*** nati_ueno has joined #openstack-infra15:54
*** reed has quit IRC15:54
jeblairmalini: i don't think so -- third party tests should be used for things that can't be tested in the cloud15:55
*** CaptTofu has quit IRC15:55
jeblairmalini: does marconi _need_ a newer version of mongo, or does it also want to support a newer version?15:56
malinijeblair: what would be agood example? load tests?15:56
sheeprinefungi: my bad, it was an MTU problem caused by a tunnel ipv6 is working fine now15:56
jeblairmalini: tests on real hardware, mostly15:56
fungimalini: or tests requiring access to specific hardware15:56
fungisheeprine: great! glad there's nothing for me to fix anyway ;)15:56
malinijeblair: marconi needs a newer mongo15:56
jeblairmalini: then that's not really a testing issue, that's a project requirements issue15:57
*** jcoufal has quit IRC15:57
jeblairmalini: openstack as a whole tries to support the latest ubuntu and fedora releases, without breaking compatibility with the latest lts and rhel releases15:57
fungimalini: does marconi not intend to target stable server distributions like the other openstack server projects do?15:57
zarofungi: my gerrit summit slides not on yet?15:57
zarofungi: you still working on that issue?15:58
fungizaro: i need to see if my fix for that job merged yet so i can re-trigger it15:58
malinifungi: 'stable server distributions' as in 'mongo that comes default with the linux distro' ?15:58
dstufftBobBall: btw i'm done with your host, thanks a ton :)15:59
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews
jeblairmalini: so generally, we run tests on the latest lts and precise (because those are the only releases that last long enough to be supported through an openstack release's life cycle15:59
fungimalini: right. the mongo which would be under security support from a stable linux distro (rhel, ubuntu lts, et cetera)15:59
jeblairlatest lts and rhel i mean15:59
malinijeblair, fungi: & precise has Mongo 2.0.4 which is pretty old :(15:59
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port
fungimalini: are you actively working with ubuntu to correct that deficiency?16:00
pleia2good morning16:00
zaropleia2: morning16:00
*** matrohon has quit IRC16:00
malinifungi: no..will tht be the correct path forward?16:00
fungimalini: it feels like something we shouldn't work around by just punting and telling users to go find their own mongo version16:00
*** dangers_away is now known as dangers16:00
jeblairmalini: yeah, if ubuntu adds a newer version to either precise or the cloud archive, we can use it; it would be good to do the same with redhat for their equivalent16:01
malinifungi: agreed..ceilometer also has the same issue, & they dont have any tests against mongo at the gate (But domebody needs to cross check me on tht)16:01
jeblairmalini: (also, for juno we will be running tests for the master branch on ubuntu trusty, but not right away)16:01
Alex_Gaynorfungi, ryanpetrello: Sadly no, no new package, I hadn't realized they'd been globally voting: false'd, sorry bout that@Q16:01
malinijeblair: what is cloud archive?16:02
fungimalini: it might be moot if marconi isn't graduating from incubation until juno, since that might allow you to only have to support running on ubuntu 14.04 and rhel 7.x16:02
ryanpetrelloAlex_Gaynor: no problem, I’m fine now that I know the whole story16:02
jeblairmalini: it's an extra archive you can add to ubuntu that has some newer versions of things required by openstack16:02
fungiAlex_Gaynor: yep, still watching the (openstack, pip, debian) bugs for new developments16:03
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Adds sanitised sourceforge source repo requirement to docs
sdaguegiven that marconi doesn't need to run in jobs with libvirt, it's probably a valid approach16:03
Alex_Gaynorfungi: sorry, I'll go ping everyone. starting with dstufft :-)16:03
jeblairmalini: zul can help with that, though he might be pretty busy right now16:03
dstufftwhat'd I do now16:03
*** mihgen has quit IRC16:03
*** jooools has quit IRC16:04
zuljeblair:  yes he is pretty busy16:04
jeblairsdague: i think we have uca installed on devstack nodes16:04
jeblairi'm going to restart nodepool nowe to pick up those two build-related changes16:05
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support detached HEAD state
fungijeblair: sdague: i thought dims was still testing that by patching it into devstack-gate16:05
jeblairfungi: er, wow, we don't add cloud archive?16:06
jeblairi thought we did that since forever16:06
malinijeblair, fungi: Trusty looks promising &Marconi is not graduating this cycle. So we can just use the sqla driver for now16:07
fungijeblair: every time we tried, we broke nova and tempest because it installs newer libvirt than we can support16:07
jeblairfungi: oh.  i guess we tried enough times i thought it stuck.16:07
*** jogo is now known as flashgordon16:07
fungiit definitely git sticky16:07
jeblairmalini: sounds like a plan; as soon as we get trusty nodes in we can start testing there16:08
fungibut yeah, at this rate we'll be running on trusty before we're able to use uca on precise i expect16:08
malinithanks jeblair & fungi16:09
jeblairflashgordon: just restarted nodepool with your 'moar servers' change and my "don't wait on error" change, hopefully it should start getting a bit more responsive16:09
malinijeblair, fungi: is there any ETA yet on when we'll Trusty at the gate?16:11
jeblairmalini: probably a few weeks after it's released, depending on when either our providers get an image in place or whether we can use glance to upload our own16:12
flashgordonjeblair: neato16:12
malinijeblair: thanks! I will check back around then16:13
fungiand the release is about 3 weeks out from now16:13
flashgordonI forgot I actually called that branch name moar ;)16:13
sdaguejeblair: no, it's definitely never really worked, which is why we're just using base precise16:13
*** ihrachys|wfh is now known as eehruchyshka|fri16:13
*** changbl has quit IRC16:14
flashgordonjeblair: at some point we need to loop back to HP and sort out why itsi failing16:14
sdagueand, honestly, given that fact that we've failed to make it work for this long, I'm cautious on any policy in using UCA going forward, because the bug fix rate doesn't seem to keep up with our needs16:14
jeblairflashgordon: i added bits to the logs and we're collecting data now16:15
*** thuc_ has joined #openstack-infra16:15
*** thuc_ has quit IRC16:15
jeblairflashgordon: we've got hpcloud 1.1 mixed in at about 200 nodes (so something like 20% of nodes should end up there)16:16
*** thuc_ has joined #openstack-infra16:16
jeblairhopefully enough to get data but not enough to slow things down considerably16:16
*** mriedem has quit IRC16:17
jeblairmordred: do you anything about a host that calls itself ""?  because it's emailing me.16:18
krotscheckI remember us tossing that name around a while ago for a dev instance, but I don’t recall it ever being set up.16:19
mgagneIs there an easy way to add yourself to reviewers without having to type your own name each time? Luckily my name isn't Nabuchodonosor but hey =)16:19
jeblairmgagne: you can use your account number16:19
jeblairmgagne: but i don't think there's an "add me" button16:20
sdaguemgagne: leave a 0 review comment16:20
mgagnejeblair: cool - I am not a number, I am a free man!16:20
flashgordonjeblair: awesome, I like how we are now using two mostly CD openstack(ish) clouds to run infra on16:20
jeblairmgagne: mordred is the new number two.16:21
jeblairmordred: i have confirmed there is a machine in our account called "" with an ipv6 address that matches the one emailing me.16:23
jeblairmordred: yet there is no storyboard-dev in dns or in puppet16:24
sdaguejeblair: from a devstack perspective, region-b looks like a straight win16:24
jeblairmordred: i suspect that this is an incomplete attempt to set up that server; can it be deleted?16:24
sdaguetough what's up with the difference between the check nodes in rax-iad -
jeblairsdague: cool.  we do spin up 30g instances there and then limit the ram to 8g in order to get enough cpu.  but i'm glad that after doing that, it works well.  :)16:25
fungisdague: bearing in mind that we're using instances with 4x the ram we need so we can get them to go faster16:25
*** markmcclain has joined #openstack-infra16:25
clarkbsdague: rax-iad are pvhvm images16:26
sdagueclarkb: ok, and the rest of rax is still not on those images?16:26
clarkbsdague: correct, we should fix that16:26
clarkbsdague: the variability in base image made us cautious16:26
sdaguefair, though no worse than all the rest of rax16:26
*** nati_ueno has quit IRC16:27
sdaguehp azs actually degraded a lot in their variability as well16:27
jeblairi think he meant the fact that the contents of the base image were actually different16:27
jeblaireg, missing packages, etc16:27
sdaguethe cffi issue tripped us up on that16:27
*** nicedice has joined #openstack-infra16:27
sdaguefwiw hp az[\d] used to have a std deviation of 3 or 4 mins, now it's up to the ~10 that rax nodes tend to have16:28
clarkbsdague: I think we are seeing the effects of a smaller pool of hypervisors16:28
sdagueas they migrate off to 1.1?16:29
clarkbjeblair: fungi: should I propose a change to switch to pvhm for rax?16:29
JayFgot a bit of an interesting test failure here. -- looks like we got a strangely bad cert on a devstack run. I think a recheck no bug will fix the error, but it's weird enough that I'm curious how it happened (see the last line of the traceback at the end of the log)16:29
clarkbsdague: yup16:29
jeblairclarkb: any chance it can wait until the label refactor patch lands? :)16:29
*** mriedem has joined #openstack-infra16:29
clarkbjeblair: oh sure16:29
clarkbjeblair: the label refactor is much more important16:30
clarkbJayF: pypi updated their certs but DNS lagged16:30
uvirtbotLaunchpad bug 1298929 in openstack-ci "certificate errors downloading pip or installing pip packages" [Undecided,New]16:30
jeblairalso "'*'"  ??16:30
sdaguejeblair: yep, inspiring, right :)16:30
fungiclarkb: they also had at least one "stuck" server with the wrong cert16:31
*** YorikSar has quit IRC16:31
clarkbjeblair: pypi was the group that stole all of those credit card numbers >_>16:31
JayFjeblair: exactly16:31
fungifastly did, i mean16:31
JayFjeblair: seeing fastly is not that disconcerting, seeing target /is/16:31
clarkbsdague: I am going to write the e-r change to switch queries over to tag filtering for file names16:31
jeblairi trust target for all of my domestic merchandise and python packaging needs!16:32
*** reed has joined #openstack-infra16:32
sdagueclarkb: ++16:32
flashgordonclarkb: ++ can you add a test to make sure we don't add in filenames  too16:32
sdaguehopefully dstufft's fix will resolve that one. It's currently our top race bug16:32
*** eehruchyshka|fri is now known as ihrachys|afk16:33
sdagueflashgordon: we're not doing any parsing of the query yet, I think that's beyond scope16:33
*** comstud is now known as bearhands16:35
*** gyee has joined #openstack-infra16:35
sdagueflashgordon: ok, I didn't realize there was syntax checking in there16:36
clarkbflashgordon: filenames are still valid16:37
clarkbflashgordon: maybe you know a bug only happens in grenade old logs16:37
clarkbflashgordon: I don't think telling everyone to stop using filename is the correct answer here16:37
fungilast logstash hit on the cert bug was more than an hour ago16:38
sdaguefungi: cool16:39
BobBalldstufft: Great! glad I could help!16:39
flashgordonclarkb: kk16:39
fungiquiet since 15:16 utc16:40
dstufftjeblair: JayF hrm, why is seeing disconcerting?16:41
fungidstufft: good point... it was in-store purchases which ended up compromised after all16:43
BobBallbtw - anyone know why nodepool is such a memory hog?  Over time it will eat up 512M of ram...16:43
BobBallI have to keep restarting it :)16:43
*** jaypipes is now known as leakypipes16:43
dstufftfungi: well it's not like has control over that certificate anyways16:43
dstufftonly fastly does16:43
*** VINOD_ has joined #openstack-infra16:43
fungidstufft: i took it as a glib joke ;)16:44
jeblairBobBall: no, i've tried and haven't found the culprit yet.  if you have any ideas, please let me know.16:44
dstufftfungi: I've had people for real tell me that the extra names on the SAN cert were some sort of security problem16:44
dstufftandi'm lik no not really :(16:44
*** HenryG has quit IRC16:44
BobBallokies - np16:44
*** hashar has quit IRC16:45
*** HenryG has joined #openstack-infra16:45
*** ameade has quit IRC16:45
jeblairBobBall: i have learned that using a lot of threads comes with a surprisingly high memory cost, but that should just end up with a higher floor, it shouldn't be a leak.16:45
*** ameade has joined #openstack-infra16:45
BobBallI'll have a poke16:46
jeblairBobBall: an earlier version had a problem with image builds using a lot of ram, but that's been fixed.16:46
jeblairnow it leaks slower.16:46
BobBallwhat was uising the ram in that case?16:46
*** VINOD_ has quit IRC16:47
*** vhoward- has left #openstack-infra16:48
*** adalbas has quit IRC16:48
jeblairBobBall: good question, i can't seem to find the commit right now16:49
*** ominakov has quit IRC16:51
*** dcramer_ has quit IRC16:52
clarkbsdague: flashgordon bug query 1285833 any idea why we can't look for that error in nova logs?16:53
clarkbit should be fine to have that there, its just odd that a message in all api logs would be an error except for nova16:54
*** zns has joined #openstack-infra16:55
sdagueclarkb: I have no idea why it's excluding n-api16:56
sdagueI'd be happy if it changed16:56
*** SumitNaiksatam has quit IRC16:56
*** adalbas has joined #openstack-infra17:00
jeblairBobBall: ah, the problem was related to keypairs -- we used to leak keypairs on failed image creations, then we ended up retrieving 40k keypair records using novaclient which used lots of ram17:00
jeblairBobBall: so that was fixed in 6ab771728ef83937db63762a848cbad476f4420517:01
jeblair(and our observed high memory usage was really a side effect of that)17:01
BobBallHmmmm - good point17:01
*** malini has left #openstack-infra17:02
BobBallso that one was just down to novaclient _using_ ram rather than leaking ram17:03
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename
clarkbsdague: flashgordon ^ proper review on that would be good though I didn't just search and replace I actually looked at each query17:04
*** khyati has joined #openstack-infra17:05
*** thuc_ has quit IRC17:06
*** markwash_ has joined #openstack-infra17:07
*** thuc has joined #openstack-infra17:07
*** markwash has quit IRC17:07
*** markwash_ is now known as markwash17:07
sdagueclarkb: sure, will do in a bit.17:07
*** thuc_ has joined #openstack-infra17:09
*** gokrokve has joined #openstack-infra17:09
*** thuc has quit IRC17:09
*** pblaho has quit IRC17:09
*** nati_ueno has joined #openstack-infra17:11
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters.
clarkband that is a follow up17:11
*** SumitNaiksatam has joined #openstack-infra17:14
*** harlowja_away is now known as harlowja17:15
*** e0ne has quit IRC17:18
sdagueclarkb: we know this works?17:19
sdagueAND tags:screen-*-api.txt17:19
*** saju_m has joined #openstack-infra17:19
*** CaptTofu has joined #openstack-infra17:22
clarkbsdague: it should, the field isn't special compared to filename17:23
*** ProfFalken has quit IRC17:23
sdaguea couple other things in there17:23
jeblairBobBall: yes, but i think it may have been holding onto that; maybe caching the info from the response...17:23
*** ProfFalken has joined #openstack-infra17:24
jeblairBobBall: which might suggest a place to investigate for the current problems..17:24
jeblairBobBall: in my local searching for a 'leak', i didn't actually spin up and destroy machines (but did nearly everything else).  perhaps novaclient is caching data from those operations, and simply the number of instances it deals with over time add up.17:25
*** changbl has joined #openstack-infra17:27
*** _dguerri has quit IRC17:28
clarkbsdague: responded. Will push a new patchset shortly17:28
*** ProfFalken has quit IRC17:28
*** Kiall has quit IRC17:29
*** Shrews has quit IRC17:29
*** vponomaryov has left #openstack-infra17:30
sdagueclarkb: great17:30
clarkbsdague: you think that should be console instead of devstacklog?17:31
*** julienvey has quit IRC17:31
clarkbsdague: and console instead of console.html?17:31
sdagueso if it's as FAIL:17:32
*** dkliban_afk is now known as dkliban17:33
sdaguewhich is matching a tempest test, then console.html is fine17:33
sdagueotherwise, we should probably default to tags:console17:33
sdaguebecause it might show up in grenade logs17:33
sdagueso 1288427.yaml - console17:34
sdague1288312.yaml - console.html17:34
*** bknudson has left #openstack-infra17:34
clarkbsdague: and 1294874.yaml should be console?17:34
*** saju_m has quit IRC17:35
*** saju_m has joined #openstack-infra17:36
*** lcheng has joined #openstack-infra17:36
sdagueclarkb: yes17:36
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters.
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename
*** moted has quit IRC17:38
*** morganfainberg_Z is now known as morganfainberg17:39
pleia2clarkb: can you log into old-wiki and get me the info I need to finish this patch? (probably just cron job and config.js)17:39
pleia2(or anyone)17:40
*** Ryan_Lane has joined #openstack-infra17:40
pleia2once this is in, bug 1127411 is the last remaining to get rid of this server17:41
uvirtbotLaunchpad bug 1127411 in openstack-ci "Move the $ redirects off of old wiki" [Medium,Triaged]
*** markmc has quit IRC17:42
clarkbpleia2: looking at the change now17:42
fungipleia2: oh, sorry, i said i'd get you that and then got sidetracked17:42
pleia2fungi: no worries, I know you've been busy :)17:43
fungii'm pretty sure everybody is ;)17:43
*** BobBall has quit IRC17:44
pleia2yeah, just trying to chip away at these servers we should get rid of17:44
clarkbpleia2: I am digging aruond on old wiki17:46
clarkbI feel so blind on this machine17:46
*** vhoward- has joined #openstack-infra17:46
*** bknudson has joined #openstack-infra17:46
clarkbpleia2: */30 * * * *17:47
clarkbfor bugdaystats/refresh_data17:47
clarkband you want config.js contents?17:47
clarkbconfig.js coming up17:48
pleia2fungi: hm, commented out?17:48
fungipleia2: seems that way17:49
fungipleia2: -rw------- 1 ttx crontab 370 Mar  7 20:54 /var/spool/cron/crontabs/ttx17:50
fungii suspect he enables or manually runs it or something?17:50
pleia2neat, so once this lands we'll actually have *working* bugdaystats too!17:50
clarkbfungi: the files in output/ are old17:50
pleia2thanks guys17:50
clarkbfungi: not sure if he is running it elsewhere maybe?17:50
clarkboh wait nevermind those should be old /me digs more for actual outpout17:51
pleia2old-wiki is definitely where status.o.o points17:51
fungiyeah, i'm thinking bugdaystats got disabled in cron a few weeks ago based on that crontab17:52
clarkbfungi: the files are in /home/ttx/bugstats17:52
clarkbfungi: and they end on march 7th17:52
clarkbso I think you are correct17:52
fungiseems confirmed17:52
*** ociuhandu has quit IRC17:52
fungipleia2: as for the other remaining task in 1127411, i think it was decided yesterday that we just delete them from dns and see who screams17:53
pleia2fungi: excellent17:54
fungibecause they've been permanent redirects for a very long time, and current documents supposedly haven't linked using those names for ages17:54
*** david-lyle is now known as david-lyle_afk17:54
fungiat least not for havana, and grizzly is now eol17:54
* fungi gets back to making it so17:55
* pleia2 nods17:55
fungibut i think i remember documentation getting cleaned up in that regard circa folsom17:55
*** krtaylor has quit IRC17:55
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Fix missing attribute error in subnodes
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add a test for subnodes
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support
jeblairrebase on master (with flashgordon's change) plus... a bugfix caught by a test!17:56
*** derekh has quit IRC17:56
pleia2hooray for tests17:56
clarkbmriedem: responded to your e-r comment17:57
mriedemclarkb: ok17:57
clarkbmriedem: basically I want to do a 1:1 mapping in that change then everyone can sweep through the queries and improve them17:57
clarkbmriedem: for example the change that is a child of that change does some of this cleanup17:57
*** dcramer_ has joined #openstack-infra17:58
clarkbjeblair: where is flashgordon's change?17:58
jeblairclarkb: merged17:58
clarkbthere is thanks17:59
clarkb*it is17:59
rossella_sjeblair: thanks for the patch!18:00
rossella_ssdague, jeblair are you around to finish the discussion we started yesterday?18:00
jeblairrossella_s: no problem; it's even passing tests now18:00
rossella_sjeblair: nice, when do you think it will be possible to see the job in action?18:02
*** mkerrin has quit IRC18:02
jeblairrossella_s: we'll need to restart zuul after it merges; so probably next week at the earliest18:03
clarkba zuul change too? /me feels like he has missed a bunch of stuff yesterday18:03
rossella_sjeblair: ok, great! I will wait then. Thanks a lot!18:03
clarkbthat reminds me I need to fix my zuul patch... not enough hours in a day18:04
clarkbjeblair: reviewing the nodepool stack now, hopefully we can get that in today18:04
*** vhoward- has left #openstack-infra18:05
sdaguehere's a fun new fail18:05
sdaguerax only18:06
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed contenteditable
*** thuc_ has quit IRC18:07
*** thuc has joined #openstack-infra18:08
jeblairmordred, sdague: it would be pretty cool if the install_pip_tarball thing didn't download from the net18:10
clarkbfungi: anteaya jeblair anyone looking at new-projects for today?18:10
jeblairclarkb: i have not18:12
ryanpetrellois there anybody in here I can nag into taking a peek at ?  I’m eager to give it a try :)18:12
*** thuc has quit IRC18:12
clarkbI want to get nodepool changes in, but I can probably look at new-projects afterwards if no one beats me to it18:12
*** thuc has joined #openstack-infra18:13
jeblairclarkb: 2/3 lgtm, i'll push a new patch for the problem i saw18:15
*** jp_at_hp has quit IRC18:16
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Create neutron-specs git repo
*** ildikov_ has quit IRC18:17
pleia2so, tripleo wants to have a local ubuntu/fedora mirror in their ci, I'm going to put the puppet stuff together to make this happen, we're not doing this for -infra, but would it be valuable for me to put it in config/ anyway if we want/need it in the future?18:17
pleia2instead of shoving it into tripleo-specific repos18:18
jeblairactually, i see another problem18:18
jeblairpleia2: i thought the point of the tripleo ci was to be part of the project infrastructure18:19
jeblairpleia2: i think that means if there are dependencies, they need to be managed that way.18:20
pleia2jeblair: well, there is still a separate tripleo-ci repo18:20
pleia2but I agree18:20
*** andreaf has quit IRC18:20
pleia2putting it in config/ makes the most sense to me18:20
fungipleia2: it seems to me like something we might legitimately want for future job stability (keeping in mind the bugs we get from time to time because of a packaging mirror going down or getting updated in wrong/non-transparent ways)18:20
pleia2fungi: yeah18:20
jeblairfungi: agreed18:20
clarkbits definitely something that once in a while we seem to agree we should have a mirror then the problems go away18:20
fungisimilar to how we might want to switch to pypi mirroring in each provider for improved latency/throughput/stability and so on18:21
clarkbjeblair: nodepool stack lgtm18:22
jeblairfungi: yep.  and dns once we can have a private network with neutron18:22
fungii'm jumping back on the nodepool changes now. i too would like to see them merge and have a weekend to bake in18:22
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add the openstack-security-notes project
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add StackTach project
clarkbjeblair: I have a comment for stacktach18:24
clarkbjeblair: did you want to address that comment?18:25
*** dims has joined #openstack-infra18:25
jeblairclarkb: not really.  feel free.18:25
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add StackTach project
clarkbthere it is18:27
clarkbI am going to give others a few minutes to review, but then I will approve the bunch so that repo creation can happen18:28
*** vkozhukalov has joined #openstack-infra18:30
*** vkozhukalov has left #openstack-infra18:30
*** jgallard has quit IRC18:31
*** signed8bit has quit IRC18:32
devanandaclarkb: hi! I'm wondering if the transifex error is either more wide spread, or a red herring. it seems to be in nova's logs too:
clarkbdevananda: I have no idea18:33
clarkbdevananda: check with the translation team maybe?18:33
*** dstanek has quit IRC18:33
clarkbdevananda: shows a diff18:34
clarkbbut thats mostly just keeping the .pot file up to date and updating .po files accordingly18:34
mordredjeblair: oh - the one in devstack?18:34
*** dstanek has joined #openstack-infra18:34
jeblairmordred: yeah18:35
mordredjeblair: well, we could certainly pre-fetch in devstack-gate18:35
jeblairmordred: it should be in the nodepool prep scripts i think18:35
clarkbdevananda: I am inclined to believe that this is a widespread problem18:36
devanandagreat :(18:36
clarkbdevananda: is there a bug yet? we can send mail to tom fifieldt and daisy to see if they know what is going on18:36
devanandano. I'll open one now18:36
devanandawait. yes18:36
uvirtbotLaunchpad bug 1298645 in ironic "translated .po files do not contain any translation text" [High,Triaged]18:37
mordredjeblair: funny enough - we DO already download in nodepool scripts18:37
clarkbdevananda: want to craft and email and send it to daisy and tom? Or do they have a list? we can send it to a list if they have one /me checks18:37
mordredjeblair: we should just put it somewhere so that we can tell destack about it :)18:37
devanandaclarkb: there's an irc channel18:37
clarkbdevananda: they do
clarkbI think I am even on that list18:37
fungiclarkb: i'm cool with all three the new-project changes, so approve at your leisure18:37
devanandai just jumped in and am pinging them18:37
clarkbfungi: danke18:38
clarkbdevananda: awesome let me know if I should find that channel too18:38
devanandaclarkb: ack18:38
anteayaclarkb: just got back, looks like all three new-projects patches are in shape to merge18:39
clarkbanteaya: yup, I just approved all of them18:39
anteayaclarkb: are you doing the manage-project honours today?18:39
anteayaanswers my question18:39
clarkbanteaya: once git01-git05 have the new repos I can run manage-projects18:39
fungimordred: agreed--though as a separate aside, it might be helpful for devstack to grow a downloads list similar to its apts/debs lists where urls for things it could conceivably want are abstracted out and can then be more easily pre-cached to a specific location it looks for18:39
jeblairmordred: cool, can you take care of that please?18:39
clarkbthat will probably happen a little after noon PST so others should feel free to trigger manage-projects while I eat lunch18:39
mordredjeblair: yes18:40
*** dizquierdo has quit IRC18:40
mordredfungi: funny story - diskimage-builder has such an abstraction18:40
fungimordred: proof that devstack should just use dib18:40
mordredfungi: it would be interesting to see if it's possible to re-use their code18:41
mordredor that18:41
jeblairfungi: how would devstack use dib?18:41
jeblairwhat would it do with the images it built?18:41
* mordred assumes fungi was joking18:41
* jeblair has been trolled18:41
fungiyes, mostly joking... though it could potentially use parts of dib without using it to build disk images directly18:41
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed #37, project UI shows edit buttons when not superuser.
openstackgerritA change was merged to openstack-infra/config: Add StackTach project
*** afazekas has joined #openstack-infra18:43
openstackgerritA change was merged to openstack-infra/config: Create neutron-specs git repo
openstackgerritA change was merged to openstack-infra/config: Add the openstack-security-notes project
*** sheeprine has left #openstack-infra18:47
*** openstackgerrit has quit IRC18:48
*** openstackgerrit has joined #openstack-infra18:48
*** dstanek has quit IRC18:49
*** thuc has quit IRC18:49
*** dguerri has joined #openstack-infra18:50
*** kiall has joined #openstack-infra18:52
*** ProfFalken has joined #openstack-infra18:54
*** thuc has joined #openstack-infra18:55
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Adds sanitised sourceforge source repo requirement to docs
*** _nadya_ has joined #openstack-infra18:58
*** _nadya_ has quit IRC18:59
*** harlowja is now known as harlowja_away19:00
*** david-lyle_afk is now known as david-lyle19:00
*** _nadya_ has joined #openstack-infra19:00
openstackgerritA change was merged to openstack-infra/storyboard: Add data load instructions to dev install doc
*** signed8bit has joined #openstack-infra19:04
*** _nadya_ has quit IRC19:05
*** e0ne has joined #openstack-infra19:07
*** wenlock_ has joined #openstack-infra19:07
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Move bugdaystats to to
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Remove unimplemented features in list view.
*** ociuhandu has joined #openstack-infra19:09
fungii'd almost forgotten about them19:10
sdaguemordred: is get-pip in nodepool, or the actual pip tar ball?19:10
*** dstanek has joined #openstack-infra19:10
pleia2fungi: probably for the best19:10
sdaguebecause if it's only get-pip, that's still got to hit the network19:11
mordredsdague: is the thing you download19:11
mordredit has everything bundled in it19:11
sdagueoh, it does?19:11
*** wenlock_ has quit IRC19:12
sdagueso just put it in the right place :)19:12
*** zehicle has joined #openstack-infra19:12
mordredthere we go!19:12
mordredwe should set use_get_pip to be default19:12
*** ociuhandu has quit IRC19:13
sdagueis it not because it's not versioned?19:13
mordredpossibly so19:13
*** CaptTofu has quit IRC19:14
sdagueand we still have a 1.5 issue, right?19:14
mordreddon't think so19:14
mordredwe  have an issue with newer tox19:14
mordredbut ...19:14
mordredlet's say two things19:14
*** CaptTofu has joined #openstack-infra19:14
mordredwe should put in logic in the tarball fork to look for the file in $FILES first19:14
clarkbmordred: sdague right pip 1.5.latest should work fine19:14
mordredthen we should have nodepool prep scripts put both and a pip tarball in $FILES19:15
clarkbit happens that making unittests use pip 1.5.latest is a pita with the state of tox19:15
*** denis_makogon_ has joined #openstack-infra19:15
sdaguemordred: yes, I'm already with you19:15 --no-index --find-links=$FILES19:15
sdagueSean Dague proposed a change to openstack-dev/devstack: cache the pip tarball
dstufftyou might like ^?19:15
*** denis_makogon_ is now known as denis_makogon19:16
*** _nadya_ has joined #openstack-infra19:16
mordreddstufft: well, we have to find get-pip.py19:17
mordreddstufft: oh - I getcha19:17
mordredthat way we can always use get-pip as the entrypoint to the process, and control the version by what's in FILES19:17
*** harlowja_away is now known as harlowja19:17
sdagueexcept where we don't have get-pip19:18 is really just like calling pip19:18
dstufftexcept with some arguments already pre-set19:18
clarkbmriedem: sdague flashgordon mtreinish re what do you guys think about my comment there? Should I fix the -1 in that change?19:18
sdaguedevstack needs to be able to bootstrap from nothing19:18
*** melwitt has joined #openstack-infra19:18
dstufftyou can embed in it if you want19:18
mordredsdague: yah - if there is no FILES/ - I think you still want to download19:19
mordredsdague: and if there is no FILES/pip*tar.gz, you want to download19:19
*** CaptTofu has quit IRC19:19
mriedemclarkb: on a call, i'll have to check it later19:19
clarkbmy ssh's in for loops say all three new projects have repos on all 5 git replicas19:19
sdagueso after my last patch we'll do both19:19
clarkbfungi: jeblair mordred SergeyLukjanov anteaya et al I am going to run manage-projects now19:19
mordredsdague: but if there is FILES/, you want to use it - and if there is FILES/pip*tar.gz - you want to do the form dstufft said above19:20
*** CaptTofu has joined #openstack-infra19:20
mordredclarkb: ++19:20
mordredsdague: cool19:20
*** CaptTofu has quit IRC19:20
*** zehicle has quit IRC19:21
*** zehicle has joined #openstack-infra19:21
clarkbit is running19:21
clarkblog is in roots homedir and dated19:21
anteayaclarkb: yay19:24
*** dcramer_ has quit IRC19:25
*** eharney has quit IRC19:26
lifelessNobodyCam: you'll get more help from derekh :>19:29
lifelessNobodyCam: but19:29
lifelessNobodyCam: see openstack-infra/tripleo-ci repository, in that look at the ci_commands file and you can grep around to see how its installed for a key19:29
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add bug for rax network blipping on get-pip
NobodyCamlifeless: Ty.. I was looking into the isse then got side tracked...19:31
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Wrap results on index.html in container div
*** packet has joined #openstack-infra19:33
sdagueclarkb: so actually it would be good to fix the think mriedem called out in -
*** ildikov_ has joined #openstack-infra19:34
sdaguebecause I think when we did the devstacklog we lost most of those hits19:34
sdagueif you change that one, then I'll +A it through after19:34
clarkbsdague: ok19:34
sdaguethen I'll run a delete of all the old bugs before we do more here, just so we aren't fixing things we might delete anyway19:35
mriedemclarkb: sdague: off my call, so i'll agree with sdague since it sounds like he's agreeing with me :)19:35
sdaguethat's slightly cheating19:35
*** jcoufal has joined #openstack-infra19:36
clarkbmanage-projects is done19:36
clarkbanteaya: want to check things?19:36
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters.
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename
clarkbmriedem: sdague ^19:37
*** dcramer_ has joined #openstack-infra19:38
* anteaya will check19:38
*** jcoufal has quit IRC19:38
anteayaI can clone all three19:40
anteayathanks clarkb19:40
*** david-lyle has quit IRC19:40
*** david-lyle has joined #openstack-infra19:40
clarkbawesome, I will look at groups and acls and add group members19:40
openstackgerritJim Rollenhagen proposed a change to openstack-infra/config: Add PyPy gate to ironic-python-agent
mriedemclarkb: so i expected all of the queries to change from like tags:"console.html" to tags:console ?19:42
clarkbmriedem: sdague: right that is what I am trying to say. That is a different change and needs to be evaluated on a per query basis19:42
clarkbthis change is just taking us from using filename to tags19:42
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: periodic cleanup of bugs
clarkbor was the initial intent at least19:43
sdaguemriedem: not always19:43
mriedemso i just got lucky on picking on the first one?19:43
sdagueit's probably worth a pass through to fix more of them19:43
sdaguebut lets do that on another round19:44
sdagueafter the deletes19:44
sdaguebecause it's pointless to fix fingerprints that are getting removed19:44
sdagueso review on the deletes would be good as well -
anteayaannegentle nkinder
nkinderanteaya: yay!19:45
annegentleanteaya: thank you!!19:45
anteayaclarkb: did the heavy lifting19:45
sdaguethat brings us down to 74 bugs that we've seen in the last 14 days19:45
sdague74 .... ooof19:45
clarkbnkinder: I was going to add you to the core group then you can add whoever you like19:46
clarkbsdague: ya that list was big :)19:46
nkinderclarkb: I was just going to ask about that, but thought I'd check if you gave me that priv. before asking19:47
sdagueI just deleted 2619:47
clarkbnkinder: done19:47
sdagueso we were at an even 10019:47
nkinderclarkb: thanks!19:48
clarkbmarkmcclain: you are the seed neutron-specs-core member19:48
clarkband sandy walsh has been added to the stacktach groups19:49
*** dstanek has quit IRC19:50
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: periodic cleanup of bugs
sdagueyay, rebase button19:50
clarkbI think that concludes new project friday19:51
clarkbACLs check out19:51
sdagueclarkb: while I'm thinking about er things19:51
anteayasandywalsh says thanks clarkb19:51
anteayawe are talking in pm he has been having difficulty with freenode rooms today19:51
sdaguethe puppet policy intentionally was set to trigger a page refresh on a commit of new queries19:52
sdaguebut I don't think that's actually happening19:52
clarkbanteaya: no problem19:52
sdagueis that just because the puppet granularity is too high?19:52
clarkbsdague: it isn't we undid it19:52
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Add link to JSON file on index.html
sdagueok, bummer19:52
clarkbsdague: running all those queries every time a change merged was quite noticeable on the cluster side19:53
clarkbgranted this was before growing the cluster19:53
clarkbsdague: so now we run them 20 minutes apart once per hour per script19:53
clarkbsdague: we can probably drop that to once every 10 minutes, twice and hour19:53
clarkbbut doing all three at the same time was sad times19:53
sdaguethat's fair19:54
openstackgerritA change was merged to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename
clarkbsdague: want me to propose a patch for every half hour?19:54
sdagueyeh, that would be great19:54
clarkber should read 10 minutes apart, once every half hour, but I think you grokked19:55
*** bookwar has quit IRC19:58
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Recreate index.html with each run
openstackgerritClark Boylan proposed a change to openstack-infra/config: Run e-r batch jobs twice an hour ten minutes apart
pleia2ok, no more bugdaystats rabbit hole19:59
clarkbsdague: ^ that should do it19:59
anteayapleia2: yay19:59
clarkbpleia2: I meant to check on tripleo-f20 today19:59
pleia2anteaya: hehe, all day has been "oh, I should fix that too"19:59
pleia2clarkb: it built! :D20:00
pleia2clarkb:     | ACTIVE | -          | Running20:00
anteayapleia2: I see that20:00
anteayapleia2: like picking up shells at the beach20:00
pleia2anteaya: right20:00
anteayathere's a nice one, oh and that one too20:00
clarkbpleia2: do we have a job to run on that node yet?20:01
* pleia2 digs up patch20:01
clarkbthanks looking20:02
clarkbhrm no derekh20:02
clarkbhonestly I think that job should be added to the layout.yaml as an experimental job /me leaves a comment20:03
pleia2he sometimes comes back in late evening ireland time, or monday20:03
* anteaya heads out for a walk20:04
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Recreate index.html with each run
*** dprince has quit IRC20:06
*** krtaylor has joined #openstack-infra20:07
*** yolanda has quit IRC20:09
*** yassine has quit IRC20:16
*** eharney has joined #openstack-infra20:23
*** _nadya_ has quit IRC20:25
*** melwitt has quit IRC20:25
*** melwitt has joined #openstack-infra20:26
*** julim has quit IRC20:27
openstackgerritBen Nemec proposed a change to openstack-infra/config: Enable cross-testing of oslo.test with dib and t-i-e
openstackgerritBen Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements
*** thuc has quit IRC20:32
*** thuc has joined #openstack-infra20:33
*** julim has joined #openstack-infra20:33
*** thuc_ has joined #openstack-infra20:35
*** zehicle has quit IRC20:36
*** thuc_ has quit IRC20:36
*** zehicle has joined #openstack-infra20:36
*** thuc_ has joined #openstack-infra20:36
*** thuc_ has quit IRC20:37
*** thuc has quit IRC20:37
*** mwagner_bbl has quit IRC20:37
*** thuc has joined #openstack-infra20:38
*** zhiyan is now known as zhiyan_20:38
*** lcostantino has quit IRC20:39
*** Sukhdev has joined #openstack-infra20:45
*** alexpilotti has quit IRC20:45
openstackgerritBen Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: update tests that should be tags:console
*** jeckersb is now known as jeckersb_gone20:50
openstackgerritBen Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements
*** CaptTofu has joined #openstack-infra20:52
*** dims has quit IRC20:53
*** andreaf has joined #openstack-infra20:54
*** _nadya_ has joined #openstack-infra20:55
*** ociuhandu has joined #openstack-infra20:56
*** julim has quit IRC20:59
*** mfer has quit IRC21:00
*** mfer has joined #openstack-infra21:00
*** mfer has quit IRC21:01
*** Ryan_Lane has quit IRC21:01
*** mestery has joined #openstack-infra21:02
sdaguettx: you still awake?21:04
*** Ryan_Lane has joined #openstack-infra21:04
*** denis_makogon has quit IRC21:07
*** amcrn has joined #openstack-infra21:08
*** jergerber has joined #openstack-infra21:08
openstackgerritA change was merged to openstack-infra/elastic-recheck: update tests that should be tags:console
*** wenlock has quit IRC21:22
*** aysyd has quit IRC21:23
openstackgerritMartin Konrad proposed a change to openstack-infra/jenkins-job-builder: SCM module: Add support for multiple Git remotes.
fungipartway through the remaining nodepool change stack. need to take a break and grab some dinner... bbiaw21:26
*** ameade has quit IRC21:26
devanandaclarkb: got a response on the i18n ML21:28
openstackgerritAndrey Kurilin proposed a change to openstack/requirements: Add junitxml to global requirements
*** markmcclain has quit IRC21:31
*** khyati has quit IRC21:36
mordredI just -1'd that ^^21:36
clarkbdevananda: roger, does that give you a path forward for ironic?21:36
clarkbmordred: its so that people can get junitxml with pypi-mirror21:37
clarkbmordred: we have created a monster21:37
devanandaclarkb: we'll try it. does it seem like a viable fix to you?21:37
clarkbdevananda: ya21:37
devanandaclarkb: great21:37
mordredclarkb: wait - no, why do they need junitxml in _our_ pypi-mirror?21:37
mordredclarkb: pypi-mirror itself takes a list of requirements repos21:38
mordredso if they're building their own mirror, they can make their own list21:38
clarkbmordred: right, but I bet they don't realize that21:38
mordredwe do not put things in global-requirements to serve mirrors, we build mirrors to serve global requirements21:38
clarkbmordred: I agree, your -1 is correct (may need to be a -2)21:39
clarkbI was mostly half jokingly commenting on why someone may want that change21:39
*** dangers is now known as dangers_away21:39
mordredclarkb: if someone on requirements-core +2/+A's that with a -1 on it it during a requirements freeze, I believe that's grounds for horsewhipping21:39
*** mriedem has quit IRC21:40
clarkbwhat is the sitrep on pypy tests?
clarkbare we still in the hack of a workaround for that?21:40
clarkbfungi: ^21:40
flashgordonis there any good way to parse rst and extract the titles?21:41
sdaguemordred: I just -2ed it21:42
clarkbflashgordon: there are a bunch of rst2stuff utilities that come with doctools21:42
clarkbflashgordon: rst2xml may make grepping easier21:42
sdaguethe best kind of -2, with no comments :)21:42
sdagueit's friday and all21:42
clarkbflashgordon: but, you should be able to to multiline sed it I would think21:42
*** khyati has joined #openstack-infra21:42
sdagueflashgordon: you maybe be over thinking things :)21:43
*** signed8bit has quit IRC21:46
*** dkranz has quit IRC21:46
clarkbalso the rst tools are python, I am sure you can import doctools and hack on the parser direclt21:46
*** rfolco has quit IRC21:47
*** _nadya_ has quit IRC21:50
*** miqui has quit IRC21:51
clarkbjeblair: fungi: where are we on nodepool?21:51
jeblairclarkb: we have sufficient +2 on the breaking change, so we could go aheand and merge that21:52
clarkbI should be around for at least another hour and a half to two hours21:53
*** thomasem has quit IRC21:53
clarkbjeblair: will defer to your judgement on this one as you are the author, do ping me if you need me to do anything to move that along21:53
jeblairclarkb: i'll merge the breaking change now; stop puppet, merge the config change, then start puppet21:54
*** khyati has quit IRC21:54
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev
clarkbjeblair: ok21:54
jeblairi don't think it's worth going further than that because we won't actually be able to use the multi stuff until the whole tree is merged21:54
*** rcleere has quit IRC21:54
clarkbjeblair: right so incorporate labels to get past the DB and config change21:54
clarkbthen let the other pieces happen more naturally21:55
clarkbjeblair: you will massage the DB too?21:55
jeblairclarkb: yep, will do21:55
*** ominakov has joined #openstack-infra21:55
*** e0ne_ has joined #openstack-infra21:55
clarkbjeblair: what is the process there? drop the entire schema or are you going to do a manual migration?21:55
jeblairclarkb: i'll probably just alter table21:55
clarkbjeblair: may be good to write that up and update the thread about the breaking change on the ml21:55
jeblairclarkb: ++21:56
*** dcramer_ has quit IRC21:57
*** e0ne has quit IRC21:57
clarkbALTER TABLE nodepool.node CHANGE image label TEXT; ? (that is mostly from memory of what the schema looks like so don't copy pasta :) )21:58
openstackgerritA change was merged to openstack-infra/nodepool: Add 'labels' as a configuration primitive
jeblairalter table node change image_name label_name varchar(255) not null;22:00
jeblairclarkb: close22:00
*** ominakov has quit IRC22:01
*** pdmars has quit IRC22:02
*** prad has quit IRC22:02
*** e0ne_ has quit IRC22:03
*** e0ne has joined #openstack-infra22:04
openstackgerritA change was merged to openstack-infra/config: Use labels in nodepool config
*** wchrisj has quit IRC22:06
*** bknudson has quit IRC22:06
jeblairok here goes22:07
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Fix image/label name typo in stats
jeblairclarkb: whoops ^; that's in production now22:11
*** mwagner_bbl has joined #openstack-infra22:11
*** dcramer_ has joined #openstack-infra22:11
clarkbjeblair: ok, will review22:11
clarkbjeblair: I also notice in the debug log the AllocationsGrantTarget lines all show 0 despite having nodes granted22:12
jeblairthat sounds problematic22:13
clarkbya not sure if just a logging bug or an actual bug yet22:13
jeblairit certainly is not building many22:14
jeblairclarkb: the nodes it is building _did_ show up in allocationgranttarget lines22:14
jeblairi lean toward the logging being correct22:14
*** khyati has joined #openstack-infra22:15
clarkbjeblair: allocation = max(allocation, 0) must be coming up as 0?22:15
clarkbjeblair: ya I think logging is correct22:15
clarkbthe __repr__ line looked good to me22:15
*** _nadya_ has joined #openstack-infra22:18
*** mrodden has quit IRC22:20
*** marun has quit IRC22:21
clarkbjeblair: I think it may be possibly for total_min_ready to be 0 (false) which causes the ration to be 0.0 and delta to be negative making max(allocation, 0) the max calculation between a negative number and 022:21
*** wenlock has joined #openstack-infra22:23
*** marun has joined #openstack-infra22:23
*** andreaf has quit IRC22:24
jeblairclarkb: are you looking in the allocator or in nodepool?22:25
clarkbjeblair: allocator22:26
clarkbjeblair: in makeAllocations22:26
*** khyati has quit IRC22:26
jeblairclarkb: but total_min_ready isn't a direct input there22:28
*** mgagne has quit IRC22:28
clarkbjeblair: ya, so over in is getNeededNodes() loop on line 1058 doing the right thing?22:28
*** SumitNaiksatam has quit IRC22:28
clarkbfor image in target.images.values(): <- specifically that loop22:29
clarkbtarget.images was the old way of doing things22:29
jeblairclarkb: i think it's something else; patch in a sec22:30
*** _nadya_ has quit IRC22:33
flashgordonsdague: I am definitly over thinking it22:33
flashgordonbut its working nicely22:34
sdagueflashgordon: yeh, I just fear putting so much effort into rules up front, vs. letting the thing naturally evolve, as I think there will be a lot of backing stuff out later22:34
sdagueor it just constraining what can be done22:35
jeblairclarkb: i can't find that line of code22:35
flashgordonsdague: perhaps22:35
clarkbjeblair: line 1062 in nodepool.py22:35
*** _nadya_ has joined #openstack-infra22:35
*** markwash_ has joined #openstack-infra22:35
flashgordonI am just looking for a list of h1 and h2  (and h3) html tags22:35
flashgordonfor starters we can gate on h1 is all there22:36
*** mrodden has joined #openstack-infra22:36
sdaguesure, it just seems like if the whole thing makes sense to a human, without one of those, it shouldn't be machine penalized22:36
clarkbjeblair: er, why is my code different than cgits?22:36
clarkbbah I didn't pull I only fetch one sec22:37
clarkbjeblair: sorry for the noise I was one commit behind22:37
*** markwash has quit IRC22:37
*** markwash_ is now known as markwash22:37
flashgordonthe concern is we may miss sections22:38
flashgordonits hard to say that doesn't make sense because section x is missing22:38
*** afazekas has quit IRC22:42
*** esp has joined #openstack-infra22:42
*** rediskin has left #openstack-infra22:43
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev
esphey folks, got a gerrit question regarding changing preferred email addresses22:44
*** Sukhdev has quit IRC22:44
mordredesp: sup/22:45
esphey mordred, this is rather silly22:45
mordredesp: I love silly22:45
espso if I change my preferred email address in launchpad, it gets reflected in gerrit correct?22:45
espbut it looks like if I started a patch using the older address22:46
espand then try to amend the commit this doesn’t work so well.22:46
openstackgerritAndrey Kurilin proposed a change to openstack/requirements: Add junitxml to global requirements
espmordred: err goes something like this committer email address <my old email> does not match your user account.22:47
mordredesp: nope - all email address information is managed in gerrit22:47
espmordred: kk22:47
mordredor - wait22:48
espmordred: let me try to register the old one in there and see if works again..22:48
mordredesp: you might need to try git commit --amend --reset-author22:48
* esp takes a sip of coffee22:48
espmordred: k, let me give it a go22:49
clarkbjeblair: now that I have the right code in my editor, I wonder if the delta is calculated wrong22:50
espphew, thanks mordred!22:50
clarkbratio * total_current is what we currently have22:50
espmordred: that did the trick.. I should write that down..22:51
*** packet has quit IRC22:51
mordredesp: w00t22:51
clarkbit doesn't really give us a desired count because is setting current from the length of number of nodes we currenlty have which makes sense22:51
esphave a good one mordred!22:52
clarkbjeblair: I think desired count should equal round(ratio * amount)22:52
mordredesp: you too!22:52
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Fix the allocation distribution
clarkbjeblair: oh wait total_current is += to amount so that is covered22:52
jeblairclarkb: ^ take a look at that...22:52
*** esp has left #openstack-infra22:52
clarkbjeblair: why would length of targets change while iterating?22:53
clarkber I don't grok why the old one didn't work but the new one looks good22:53
clarkbwe basically divide allotments based on what hasn't been alloted rather than the global scope of things22:54
jeblairclarkb: yeah -- if you look at the earlier change, line 244:
jeblairclarkb: that had a similar effect of ensuring that by the end of the list, ratio was 1.022:55
jeblairclarkb: so this change tries to do the same thing22:55
jeblairclarkb: (we could get something similar by removing the line "total_current -= agt.request_target.current" which has the advantage of scaling the ratio in less weird ways, but i don't think it would converge on 1.0 in that case, so you could still end up dropping nodes)22:56
clarkbthis way eventually your ratio goes to 122:56
clarkbwhich may mean one jenkins master gets slightly more nodes22:57
clarkbbut should be ok in the big picture of things22:57
*** thuc has quit IRC22:57
*** jergerber has quit IRC22:57
jeblairyep.  it doesn't behave quite as well in the case of massive differences in node numbers as i would like (it will eventually settle, but it might do some pretty big swings along the way).  but i think that's what we had before.22:58
jeblairso i'm aiming for getting that back, and maybe with more unit tests we can tweak it later22:58
jeblairand fwiw, that change still passes the allocator unit tests i wrote so far22:59
*** thuc_ has joined #openstack-infra23:00
jeblairokay, restarting with those 2 changes now23:01
jeblairit has, in fact, decided to launch many many nodes.23:01
jeblairwhich, on balance, is probably better than none23:01
jeblairthe new allocator output looks good23:03
clarkbException: Server 315d8373-e841-4a3b-b0b7-5a86d62288ef for node id: 3305532 status: ERROR <- is that the hpcloud east failures?23:04
jesusaurusprobably not. the east failures are packet loss23:05
jeblairClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-83b693f2-b1c2-4cfe-9813-7658c2f470dc)23:05
jeblairis that? ^23:05
clarkbjeblair: right if a node doesn't get networkign wouldn't it go itno an error state?23:06
*** SlickNik is now known as gerrit23:06
jeblairclarkb: i dunno?23:06
*** paul-- has joined #openstack-infra23:06
clarkbjeblair: sorry that was for jesusaurus23:06
clarkbtab complete failed me23:06
*** gerrit is now known as Guest3549423:06
*** Guest35494 is now known as SlickNik23:07
*** zns has quit IRC23:07
jesusaurusclarkb: no, it would be running and the nic would be linked, but not all packets would make it to the other end23:07
jesusaurusso, from a nova perspective, everything would look fine, not ERROR23:08
clarkbjesusaurus: fwiw we have seen other issues like a high error rate in east23:08
clarkbthat apparently are nto related to the networking23:08
jesusaurusmight be worth mentioning to support if you're seeing that23:08
jeblair| tripleo-test-cloud | reviewstats | 7847395e-c9d6-40d5-b880-050e70c9151d | None |23:08
jeblairwhat's that?23:08
clarkbjeblair: I think lifeless was going to host reviewstats on tripleo-test-cloud but russellb resurrected it23:10
clarkbjeblair: I assume you are looking at an alien list23:10
clarkband ideally our tenant wouldn't be used for things like that23:10
lifelessjeblair: oh yeah, delete that23:11
jeblairlifeless: please don't host random stuff in the openstack-infra tenant of the tripleo test cloud23:11
lifelessjeblair: sure, sorry. te-broker is currently in that category23:11
lifelessjeblair: its not however random23:11
*** CaptTofu has quit IRC23:12
jeblairlifeless: yeah23:12
lifelessjeblair: you have two tenants in the cloud btw - nodepool and infra23:12
*** CaptTofu has joined #openstack-infra23:12
*** dhellmann is now known as dhellmann_23:13
openstackgerritA change was merged to openstack-infra/nodepool: Fix image/label name typo in stats
jeblairlifeless: ack; i see creds for both.  i was looking at the nodepool one.23:14
*** david-lyle has quit IRC23:16
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Lower tripleo max-servers to 35
*** CaptTofu has quit IRC23:16
jeblairrestarted puppet on nodepool23:21
openstackgerritA change was merged to openstack-infra/nodepool: Fix the allocation distribution
*** melwitt has quit IRC23:26
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
openstackgerritA change was merged to openstack-infra/config: Lower tripleo max-servers to 35
*** gyee has quit IRC23:31
*** zns has joined #openstack-infra23:36
*** pcrews_ has joined #openstack-infra23:39
*** sandywalsh has joined #openstack-infra23:40
*** pcrews has quit IRC23:42
*** thuc_ has quit IRC23:44
*** CaptTofu has joined #openstack-infra23:44
*** thuc has joined #openstack-infra23:44
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Upgraded jasmine
*** thuc has quit IRC23:49
*** gokrokve has quit IRC23:49
*** gokrokve has joined #openstack-infra23:50
*** jhesketh_ has quit IRC23:54
*** julim has joined #openstack-infra23:54
*** gokrokve has quit IRC23:54
*** leakypipes has quit IRC23:55
*** leakypipes has joined #openstack-infra23:55
fungisorry, got pulled into more packing/cleaning after dinner, back now23:57
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added unit tests for string_util.
fungiclarkb: pypy situation is Alex_Gaynor is circling back around to find out whether we can get fixed/updated pypy packages which solve the issue, i think. no new movement on openstack/pip/debian bug reports recently23:58
fungiclarkb: working through the rest of the nodepool docs completion change now23:58
*** homeless has quit IRC23:58
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master

Generated by 2.14.0 by Marius Gedminas - find it at!