Friday, 2014-03-21

*** saschpe has quit IRC00:00
clarkbjeblair: before you go I will have a comment for the second change too00:00
clarkbjust need to finish reviewing the whole change00:00
*** nati_ueno has quit IRC00:01
*** MarkAtwood has quit IRC00:02
*** miqui has quit IRC00:03
fungijeblair: new py3k-precise images are working like a charm...
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Increase the default pip socket timeout
*** rcleere has joined #openstack-infra00:03
derekhclarkb: ^^ thanks for the pointer to the correct place to set it00:04
*** homeless has quit IRC00:06
*** moted has joined #openstack-infra00:06
*** saschpe has joined #openstack-infra00:07
clarkbjeblair: and commented00:07
clarkbderekh: approved, we will need to wait for new images to see that everywhere but that should happen tonight assuming zuul merges that change quickly enough for us00:08
derekhclarkb: ok, thanks00:08
clarkbfungi: quick question about the py3k pip fix. how does pip know it should end up in the python3 python path?00:10
*** jhesketh__ has joined #openstack-infra00:11
*** azneita has joined #openstack-infra00:11
*** sabari has quit IRC00:11
clarkbfungi: I guess it ends up in /usr/local/bin/pip and will properly run under any python version00:11
*** rhsu has quit IRC00:12
*** SumitNaiksatam has quit IRC00:15
*** signed8bit has quit IRC00:16
*** rwsu has quit IRC00:18
*** CaptTofu has joined #openstack-infra00:19
*** Sukhdev has quit IRC00:20
*** markmcclain has joined #openstack-infra00:21
*** sandywalsh has joined #openstack-infra00:22
*** rpodolyaka1 has joined #openstack-infra00:23
*** _Alexandra_ has joined #openstack-infra00:23
*** jgrimm has joined #openstack-infra00:23
*** mrodden has quit IRC00:23
*** markmcclain1 has joined #openstack-infra00:24
*** azneita has quit IRC00:24
*** markmcclain has quit IRC00:26
*** rpodolyaka1 has quit IRC00:27
openstackgerritA change was merged to openstack-infra/bugdaystats: Add tox.ini, and do syntax cleanup
*** sarob has joined #openstack-infra00:27
*** sarob_ has joined #openstack-infra00:28
*** bhuvan has quit IRC00:28
*** _Alexandra_ has quit IRC00:30
*** thuc has quit IRC00:31
*** sarob has quit IRC00:31
*** thuc has joined #openstack-infra00:32
clarkbjeblair: re comments on the subnode change. Do you think we should ignore that problem for now and take the subnode change as is? Or should we/you/someone rebase your change atop a config fix?00:34
*** rcleere has quit IRC00:34
clarkbI am going to mock up what I thought would fix it in my comments00:35
*** thuc has quit IRC00:35
*** _Alexandra_ has joined #openstack-infra00:36
marunclarkb: feel free to approve :)
clarkbjeblair: one more comment00:40
clarkbmarun: is it running and passing now?00:40
*** SumitNaiksatam has joined #openstack-infra00:41
clarkbmarun: if not would it make more sense to keep it as an experimental job until it is somewhat reliable?00:41
*** esker has joined #openstack-infra00:41
marunclarkb: it's running and passing now00:43
clarkbgreat /me approves00:43
marunclarkb: it's a tiny subset of the unit tests that I've split off.00:43
*** rhsu has joined #openstack-infra00:43
marunclarkb: the hard part was simply getting tox to execute without that permissions error, but the tests themselves are fine00:43
*** khyati has quit IRC00:46
*** marun has quit IRC00:47
openstackgerritA change was merged to openstack-dev/pbr: Fixes blocking issue on Windows
*** wchrisj has quit IRC00:50
*** esker has quit IRC00:50
*** mbacchi has quit IRC00:51
openstackgerritA change was merged to openstack-infra/config: Add neutron functional job to check (non-voting)
*** david-lyle has joined #openstack-infra00:54
*** yongli has quit IRC01:02
*** yongli has joined #openstack-infra01:04
jeblairclarkb: i responded again, but i also want to think about some other refactoring...01:04
*** wchrisj has joined #openstack-infra01:05
jeblairclarkb: in particular, whether it's worth a bit of config refactor to be able to say "build 3 node clusters out of devstack-precise images and label them 'multinode-precise'".01:05
jeblairclarkb: that might be a solution to the thing you pointed out and also be more flexible (we could have fewer image types)01:05
*** sabari has joined #openstack-infra01:09
openstackgerritA change was merged to openstack-infra/config: Increase the default pip socket timeout
*** jamielennox has quit IRC01:11
*** rcleere has joined #openstack-infra01:14
clarkbjeblair: after that last comment I am actually +2 on the change as is if we document the limitation01:18
clarkbjeblair: but do also agree that the config could be made to be more natural01:18
*** markmcclain1 has quit IRC01:19
* gtt116 is away: 我很忙01:20
clarkbgtt116: please disable your away notice01:22
*** jamielennox has joined #openstack-infra01:22
*** rpodolyaka1 has joined #openstack-infra01:23
*** rcleere has quit IRC01:24
*** changbl has joined #openstack-infra01:27
*** rpodolyaka1 has quit IRC01:28
*** krotscheck has quit IRC01:37
*** krotscheck has joined #openstack-infra01:38
*** mgagne has quit IRC01:38
*** MarkAtwood has joined #openstack-infra01:38
*** MarkAtwood has quit IRC01:39
*** thuc has joined #openstack-infra01:42
*** yaguang has joined #openstack-infra01:42
*** NobodyCam_ has joined #openstack-infra01:42
*** sarob_ has quit IRC01:42
*** sarob has joined #openstack-infra01:43
*** NobodyCam_ has quit IRC01:43
*** krotscheck has quit IRC01:44
*** derekh has quit IRC01:44
*** thuc has quit IRC01:46
*** sarob has quit IRC01:47
*** sabari has quit IRC01:47
*** sarob has joined #openstack-infra01:52
*** BadCub01 has joined #openstack-infra01:52
*** sarob_ has joined #openstack-infra01:53
*** thuc has joined #openstack-infra01:53
*** wchrisj has quit IRC01:54
*** stack555 has quit IRC01:54
*** gokrokve has joined #openstack-infra01:55
*** sarob has quit IRC01:56
*** thuc_ has joined #openstack-infra01:57
*** thuc has quit IRC01:57
*** thuc has joined #openstack-infra01:58
*** mriedem has quit IRC02:00
*** thuc_ has quit IRC02:02
*** amcrn has quit IRC02:02
*** yjiang has joined #openstack-infra02:02
*** wchrisj has joined #openstack-infra02:04
StevenKIs nodepoll having issues? The graphs show a very large number of available nodes.02:07
clarkbStevenK yup looks dead. I am eating dinner now can look in about an hour02:11
clarkbjeblair fungi ^02:11
clarkbmordred ^02:11
jeblairclarkb: ack02:11
*** wchrisj has quit IRC02:12
jeblairif i had to guess, i'd think it might have something to do with region b02:12
*** mriedem has joined #openstack-infra02:13
*** bhuvan has joined #openstack-infra02:17
*** jepoy_ has quit IRC02:17
jeblairit's starting to look like the problem is that nodepool is not seeing zmq events from jenkinses02:17
jeblairso it never sees builds start or end02:18
*** pcrews has quit IRC02:21
*** packet has quit IRC02:21
*** bhuvan has quit IRC02:22
*** rpodolyaka1 has joined #openstack-infra02:24
jeblairthose threads are just sitting waiting for something from zmq02:25
jeblairclarkb: so it's probably some variant of the "zmq stops sending/receiving data for no apparent reason and with no error reported" issue we've seen before02:26
*** _Alexandra_ has quit IRC02:26
clarkbwas there a network burp? usually that comes along with services going away or the network dying02:26
*** rpodolyaka1 has quit IRC02:28
*** jepoy has joined #openstack-infra02:29
jeblairclarkb: oh, i think rax rebooted it02:29
jeblairi think we still have a memory leak (just a much slower one now)02:29
jeblairi'll stop nodepool mark all nodes for deletion, and start again02:31
jeblair(and hope zmq has better luck this time)02:31
*** saschpe has quit IRC02:31
clarkbjeblair you think rax rebooted the nodepool node?02:32
jeblairclarkb: yes, uptime and swap graphs suggest that02:32
*** saschpe has joined #openstack-infra02:33
*** markwash has quit IRC02:33
*** markwash has joined #openstack-infra02:37
*** thuc_ has joined #openstack-infra02:39
*** thuc has quit IRC02:39
jeblairokay, i added a job named 'test' to all 7 dynamic jenkins masters and made sure nodepool received updates from all of them02:39
jeblairnodepool is deleting all 782 nodes now02:42
jeblairbeing able to do that with "update node set state=4" is nice.  :)02:44
*** vkozhukalov has joined #openstack-infra02:45
clarkboh right since delete is decouple from the state change02:46
clarkbyou can "delete" everything via mysql02:46
jeblairthe first new nodes are being used now02:47
jeblairthere were still a few jobs running, so it's likely that nodepool will yank the nodes out from under them which will cause a gate reset02:48
jeblairbut we should lose at most one change02:48
jeblairStevenK: thanks!02:49
*** mgagne has joined #openstack-infra02:51
*** mwagner_notHere has joined #openstack-infra03:00
jeblairclarkb, fungi: i cleaned up leaked rax nodes03:01
*** mwagner_lap has joined #openstack-infra03:02
jeblairlifeless: OverLimit: Quota exceeded for ram: Requested 16384, but already used 944128 of 960000 ram (HTTP 413) (Request-ID: req-ba4579c4-ad5d-447f-a285-446a92bfe464)03:03
jeblairlifeless: i'm seeing that in nodepool logs, though there are only 41 instances, which should use significantly less than 944128 kb03:04
jeblairStevenK: ^03:04
*** jp_at_hp has quit IRC03:04
lifelessjeblair: at a guess this nova quota bug is more serious than we thought03:05
lifeless| memory_mb_used       | 703488 |03:05
jeblairthat's not urgent atm, and all the lines on the graph are going in the right directions, so i'm going to resume being afk03:07
StevenKjeblair: Thanks for fixing03:08
*** mwagner_lap has quit IRC03:08
*** jepoy has quit IRC03:09
*** mriedem has quit IRC03:10
*** krtaylor has joined #openstack-infra03:11
*** jepoy has joined #openstack-infra03:21
*** rpodolyaka1 has joined #openstack-infra03:25
*** thuc_ has quit IRC03:29
*** rpodolyaka1 has quit IRC03:29
*** thuc has joined #openstack-infra03:30
*** zehicle_at_dell has quit IRC03:31
*** zehicle_at_dell has joined #openstack-infra03:32
*** thuc has quit IRC03:34
*** CaptTofu has quit IRC03:41
*** CaptTofu has joined #openstack-infra03:41
*** pmathews has joined #openstack-infra03:43
*** jhesketh_ has quit IRC03:44
*** CaptTofu has quit IRC03:46
*** zhiyan_ is now known as zhiyan03:50
*** pmathews has quit IRC03:56
*** pmathews has joined #openstack-infra03:57
*** alex-cafehackwla is now known as _alexandra_04:03
*** packet has joined #openstack-infra04:06
*** MarkAtwood has joined #openstack-infra04:07
*** alff has joined #openstack-infra04:13
lifelessjeblair: for future reference04:14
uvirtbotLaunchpad bug 1284424 in tripleo "nova quota statistics can be incorrect" [High,Triaged]04:14
*** thuc_ has joined #openstack-infra04:21
*** rpodolyaka1 has joined #openstack-infra04:26
*** rpodolyaka1 has quit IRC04:30
*** wchrisj has joined #openstack-infra04:30
*** khyati has joined #openstack-infra04:35
*** ArxCruz has quit IRC04:36
*** morganfainberg is now known as morganfainberg_Z04:39
*** sarob_ has quit IRC04:40
*** masayukig has quit IRC04:43
*** morganfainberg_Z is now known as morganfainberg04:44
*** jhesketh_ has joined #openstack-infra04:48
*** pmathews has quit IRC04:48
*** MarkAtwood has quit IRC04:49
SergeyLukjanovjeblair, clarkb, fungi, mordred, I'm sorry, I have tons of internal meetings, so, quite inactive in IRC this week and probably will be such till the mid next week. Trying to make up reviews.04:49
*** markwash has quit IRC04:53
*** jdurgin1 has quit IRC04:58
SergeyLukjanovttx, sdague, yay, it was merged, looks like stable should start working again05:08
openstackgerritA change was merged to openstack-infra/devstack-gate: Stop printing network diagnostics
*** jdurgin has joined #openstack-infra05:11
*** mgagne has quit IRC05:11
*** gokrokve has quit IRC05:12
*** talluri has joined #openstack-infra05:12
*** gokrokve has joined #openstack-infra05:12
*** sabari has joined #openstack-infra05:13
*** vogxn has joined #openstack-infra05:15
*** gokrokve has quit IRC05:17
*** thuc_ has quit IRC05:22
*** khyati has quit IRC05:22
*** thuc has joined #openstack-infra05:23
*** rpodolyaka1 has joined #openstack-infra05:25
*** thuc has quit IRC05:27
*** marun has joined #openstack-infra05:29
*** gyee has quit IRC05:30
*** rpodolyaka1 has quit IRC05:30
*** rpodolyaka1 has joined #openstack-infra05:31
*** packet has quit IRC05:34
*** vogxn1 has joined #openstack-infra05:36
*** vogxn has quit IRC05:38
openstackgerritA change was merged to openstack-infra/config: Use correct path to puppetboard vhost.
*** CaptTofu has joined #openstack-infra05:42
*** gokrokve has joined #openstack-infra05:43
*** CaptTofu has quit IRC05:46
*** rhsu has quit IRC05:49
*** mihgen has joined #openstack-infra05:50
openstackgerritKazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge
*** rhsu has joined #openstack-infra05:51
*** alff has quit IRC05:54
*** gokrokve has quit IRC05:56
*** mrda is now known as mrda_weekend05:58
openstackgerritKazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge
*** pmathews has joined #openstack-infra06:01
*** mihgen has quit IRC06:01
*** wchrisj has quit IRC06:02
*** gokrokve has joined #openstack-infra06:04
*** lttrl has quit IRC06:07
mordrednibalizer: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid relationship: File[/puppetboard/] { require => User[undef] }, because User[undef] doesn't seem to be in the catalog06:08
mordredI have NO IDEA why it's doing that06:08
*** gokrokve has quit IRC06:10
SergeyLukjanovmordred, morning :)06:10
SpamapSIs the gate .. 9 hours behind?06:13
SergeyLukjanovSpamapS, yup06:13
SergeyLukjanovSpamapS, but it looks like it's going to be better06:14
*** mihgen has joined #openstack-infra06:15
*** ildikov_ has quit IRC06:16
*** vkozhukalov has quit IRC06:16
*** _alexandra_ is now known as alex-gone06:27
*** ociuhandu has quit IRC06:34
*** _nadya_ has joined #openstack-infra06:35
*** rwsu has joined #openstack-infra06:35
*** _nadya_ has quit IRC06:45
*** mrmartin has joined #openstack-infra06:45
*** rpodolyaka1 has quit IRC06:46
*** zhiwei has joined #openstack-infra06:49
*** vogxn1 has quit IRC07:03
*** gokrokve has joined #openstack-infra07:06
*** lukego has joined #openstack-infra07:10
*** gokrokve has quit IRC07:10
*** jamielennox is now known as jamielennox|away07:12
*** mihgen has quit IRC07:16
*** mihgen has joined #openstack-infra07:18
*** mihgen has quit IRC07:22
*** ildikov_ has joined #openstack-infra07:23
*** jepoy_ has joined #openstack-infra07:26
*** mkoderer has joined #openstack-infra07:28
*** jepoy has quit IRC07:29
*** _nadya_ has joined #openstack-infra07:31
*** mflobo_ has quit IRC07:32
*** _nadya_ has quit IRC07:36
*** tnurlygayanov has quit IRC07:37
*** tnurlygayanov has joined #openstack-infra07:39
*** mrmartin has quit IRC07:40
openstackgerritA change was merged to openstack/requirements: Removed openstacksdk from projects.txt
*** CaptTofu has joined #openstack-infra07:43
*** e0ne has joined #openstack-infra07:43
*** mwagner_notHere has quit IRC07:43
*** mwagner_notHere has joined #openstack-infra07:44
*** yolanda has joined #openstack-infra07:44
*** nati_ueno has joined #openstack-infra07:47
*** CaptTofu has quit IRC07:47
*** nati_ueno has quit IRC07:48
*** afazekas has quit IRC07:49
*** nati_ueno has joined #openstack-infra07:50
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Tidy up app.js - limit lines to 80chars
*** vkozhukalov has joined #openstack-infra07:57
*** sabari has quit IRC07:57
*** mihgen has joined #openstack-infra08:01
*** Ryan_Lane has quit IRC08:04
*** gokrokve has joined #openstack-infra08:05
openstackgerritKazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge
*** Ryan_Lane has joined #openstack-infra08:06
*** flaper87|afk is now known as flaper8708:07
openstackgerritIvan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves
*** gokrokve has quit IRC08:09
*** hashar has joined #openstack-infra08:09
*** e0ne has quit IRC08:11
openstackgerritThomas Goirand proposed a change to openstack-infra/git-review: git-review.1 manpage fix for groff warnings
*** rpodolyaka1 has joined #openstack-infra08:14
*** mwagner_notHere has quit IRC08:14
*** mwagner_notHere has joined #openstack-infra08:15
*** ominakov has joined #openstack-infra08:16
*** mrmartin has joined #openstack-infra08:17
*** e0ne has joined #openstack-infra08:18
*** rpodolyaka1 has quit IRC08:18
*** afazekas has joined #openstack-infra08:20
openstackgerritA change was merged to openstack-infra/devstack-gate: Update Readme to setup jenkins accout
openstackgerritAleksey Chuprin proposed a change to openstack-infra/config: Add gate-magnetodb-devstack job
*** jgallard has joined #openstack-infra08:26
openstackgerritAleksey Chuprin proposed a change to openstack-infra/config: Add gate-magnetodb-devstack job
*** pradnya has joined #openstack-infra08:31
pradnyaworking on bug id: Bug #1201201  want to submit doc for same please  help08:33
uvirtbotLaunchpad bug 1201201 in tempest "Document how to install tempest without devstack" [Wishlist,Confirmed]
*** mrmartin has quit IRC08:43
*** jcoufal has joined #openstack-infra08:43
*** mrmartin has joined #openstack-infra08:45
*** vogxn has joined #openstack-infra08:50
*** jp_at_hp has joined #openstack-infra08:50
*** ihrachys is now known as ihrachys|wfh|tax08:55
*** ildikov_ has quit IRC09:03
*** gokrokve has joined #openstack-infra09:05
*** gokrokve has quit IRC09:10
*** mrmartin has quit IRC09:12
*** derekh has joined #openstack-infra09:13
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets
*** jcoufal has quit IRC09:17
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Special case django_openstack_auth in translation jobs
*** andreaf has joined #openstack-infra09:18
*** jcoufal has joined #openstack-infra09:21
*** jcoufal has quit IRC09:21
*** dizquierdo has joined #openstack-infra09:21
*** nati_ueno has quit IRC09:29
*** CaptTofu has joined #openstack-infra09:31
*** pblaho has joined #openstack-infra09:32
*** jooools has joined #openstack-infra09:32
*** alff has joined #openstack-infra09:33
*** dizquierdo has quit IRC09:43
*** morganfainberg is now known as morganfainberg_Z09:48
*** ildikov_ has joined #openstack-infra09:48
*** nati_ueno has joined #openstack-infra09:49
*** pblaho has quit IRC09:54
*** saju_m has joined #openstack-infra09:55
*** jepoy_ has quit IRC09:55
*** rlandy has joined #openstack-infra09:57
*** jcoufal has joined #openstack-infra10:05
*** david-lyle has quit IRC10:05
*** gokrokve has joined #openstack-infra10:06
*** yjiang has quit IRC10:07
*** pblaho has joined #openstack-infra10:08
*** gokrokve has quit IRC10:10
*** _nadya_ has joined #openstack-infra10:14
*** julienvey has joined #openstack-infra10:14
*** rpodolyaka1 has joined #openstack-infra10:15
*** rpodolyaka1 has quit IRC10:19
*** _nadya_ has quit IRC10:24
*** yaguang has quit IRC10:25
*** vogxn has quit IRC10:26
*** jlibosva has joined #openstack-infra10:31
*** ominakov_ has joined #openstack-infra10:34
*** ominakov_ has quit IRC10:34
*** ildikov_ has quit IRC10:36
*** markmc has joined #openstack-infra10:37
mordredthe gate seems to be caught up10:41
*** CaptTofu has quit IRC10:43
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: update bug to include the cleanup case
*** zhiwei has quit IRC10:47
*** pblaho has quit IRC10:48
*** jcoufal has quit IRC10:49
*** dizquierdo has joined #openstack-infra10:52
*** hashar has quit IRC10:55
*** nati_ueno has quit IRC10:58
*** nati_ueno has joined #openstack-infra10:58
SergeyLukjanovmordred, yay!11:01
SergeyLukjanovmordred, I've not seen such gate queue size for a long time ;)11:01
*** nithya_g has joined #openstack-infra11:01
*** nati_ueno has quit IRC11:02
*** gokrokve has joined #openstack-infra11:07
*** gokrokve has quit IRC11:11
*** rpodolyaka1 has joined #openstack-infra11:16
dimsSergeyLukjanov, trouble last night on rax nodes?11:17
*** johnthetubaguy has joined #openstack-infra11:19
*** rpodolyaka1 has quit IRC11:20
*** vkozhukalov has quit IRC11:21
*** saju_m has quit IRC11:22
*** vkozhukalov has joined #openstack-infra11:23
openstackgerritAndrea Frittoli  proposed a change to openstack-infra/devstack-gate: Support Keystone V3 based gate
*** e0ne has quit IRC11:30
sdaguemordred: yeh, we killed the top 2 bugs yesterday morning, and there were fixes for a couple others in the queue during the day11:32
pradnyaHi sdague, I am working on bug id: Bug #1201201  want to submit doc for same please  help11:35
uvirtbotLaunchpad bug 1201201 in tempest "Document how to install tempest without devstack" [Wishlist,Confirmed]
sdaguepradnya: cool11:35
sdaguewhat help do you need?11:35
pradnyaI have created the installation document for tempest but i dont know where exactly to submit11:37
pradnyawhether it should be in .odt format or .rst format11:37
openstackgerritAndrea Frittoli  proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline
pradnyai am following the link
*** saju_m has joined #openstack-infra11:38
pleia2good morning11:38
pradnyareached till the step 311:38
pradnyahere i dont see any branch for tempest11:39
pradnyahow should i proceed now11:39
*** VINOD_ has joined #openstack-infra11:43
sdaguepradnya: so for this it should be in .rst11:43
*** salv-orlando has quit IRC11:43
sdagueand you should submit it to the tempest tree11:43
sdaguesomewhere in there11:44
pradnyaokay.  I have .rst file with me. Should i submit it in tempest code branch itself?11:44
pradnyawhile cloning i will use the link is it correct?11:45
sdaguepradnya: no, just clone tempest11:46
sdaguewhich will be
sdaguethe doc/source is the directory11:46
sdaguethe new rst file(s) should go there. I think once we see the first patch we might suggest file changes and such11:47
sdaguethat set of documents gets published here - once things are committed11:47
pradnyaokay. I will submit it in
*** zhiwei has joined #openstack-infra11:53
*** jepoy has joined #openstack-infra11:55
*** jhesketh_ has quit IRC11:56
*** jhesketh__ has quit IRC11:56
*** jgallard has quit IRC11:58
*** ildikov_ has joined #openstack-infra12:01
*** gokrokve has joined #openstack-infra12:07
*** pdmars has joined #openstack-infra12:08
*** _nadya_ has joined #openstack-infra12:10
*** gokrokve has quit IRC12:12
*** weshay has joined #openstack-infra12:14
sdaguepradnya: great12:14
*** salv-orlando has joined #openstack-infra12:14
pradnyaThanks sdague!! I have submitted the .rst file12:15
*** e0ne has joined #openstack-infra12:15
sdaguecool, thanks, I'll try to review later today12:16
*** ArxCruz has joined #openstack-infra12:16
*** rpodolyaka1 has joined #openstack-infra12:16
sdagueSergeyLukjanov: the important thing about the gate being clear again is I like having the gate to myself in the morning :)12:16
sdaguebecause no one else is +Aing code in my morning12:17
*** _nadya_ has quit IRC12:17
*** e0ne has quit IRC12:17
*** e0ne has joined #openstack-infra12:18
openstackgerritBjörn Michaelsen proposed a change to openstack-infra/git-review: add query option
pradnyaHi sdague, I am seeing multiple failures in Jenkins for  for patchset 312:18
pradnyabut in patchset 1, there was success for the same12:18
fungisdague: i could fix that for you, you know ;)12:18
pradnyaIs there any specific reason for those failures12:19
*** e0ne has quit IRC12:19
*** e0ne has joined #openstack-infra12:19
*** alexpilotti has joined #openstack-infra12:20
*** aysyd has joined #openstack-infra12:20
*** rpodolyaka1 has quit IRC12:20
fungipradnya: if you look at the pep8 job log, that shows you have a syntax error12:22
fungipradnya: it's probably the reason basically all the jobs failed12:22
*** mriedem has joined #openstack-infra12:23
*** e0ne has quit IRC12:24
*** _nadya_ has joined #openstack-infra12:28
*** mwagner_notHere is now known as mwagner_wfh12:28
*** mbacchi has joined #openstack-infra12:28
*** _nadya_ has quit IRC12:29
*** e0ne has joined #openstack-infra12:30
openstackgerritBjörn Michaelsen proposed a change to openstack-infra/git-review: add query option
*** yolanda has quit IRC12:32
*** e0ne_ has joined #openstack-infra12:33
*** _nadya_ has joined #openstack-infra12:33
*** e0ne has quit IRC12:34
* anteaya loves how sdague owns the morning gate12:38
sdaguemy precious12:38
SergeyLukjanovsdague, :)12:38
SergeyLukjanovsdague, will we add sahara to the integrated gate someday in Juno?12:39
fungiSergeyLukjanov: that happens right around the same time as sahara graduates from incubation12:40
SergeyLukjanovfungi, already graduated :)12:40
fungioh! all's the better12:40
SergeyLukjanovfungi, but will be part of J release, so...12:40
*** russellb is now known as rustlebee12:40
fungiwill it be part of the integrated release of icehouse, or not until juno?12:40
SergeyLukjanovfungi, J12:41
*** dims has quit IRC12:41
fungiright, so it should probably happen right after icehouse is released, or at least shortly after the juno summit wraps up12:41
fungibut i would say as soon as people want to review/approve the patches for it after icehouse releases is fair game12:41
anteayafungi: is there an etherpad yet for new project creation for today?12:42
sdagueSergeyLukjanov: so once we're back on open master12:43
fungianteaya: no, but we didn't do one last week either because there were only maybe four to work through12:43
sdagueand the stable trees are all cut12:43
*** thomasem has joined #openstack-infra12:43
anteayafungi: so doing one today, or no?12:43
pleia2fungi: if you have some time, can you look at how bugdaystats is configured on old-wiki? I need the cron job and config.js12:44
SergeyLukjanovsdague, oh, sure, we could enable sahara by default in d-g for master12:44
sdagueand there is actually a real heat race bug today!12:44
fungianteaya: if the list is half a dozen or more, then coordinating it through the etherpad might help, but last time we basically just merged them all at the same time, fixed any merge conflicts, then once they were in and appearing on all the git servers as empty repos we ran manage-projects once and that was pretty much it12:44
*** CaptTofu has joined #openstack-infra12:44
openstackgerritRicardo Carrillo Cruz proposed a change to openstack-infra/config: Initial openstackdroid commit for Stackforge, with review changes
sdagueSergeyLukjanov: yeh, once the stable/icehouse branches are all out and we have that selection logic in there12:45
fungidid some quick spot checks afterward to make sure it was all sane... not much to it12:45
pleia2anteaya: browsing reviews, should we bring the UTC comment from fungi to the -dev list for discussion?
fungipleia2: sure, just a few minutes and i'll take a look12:45
pleia2fungi: thank you12:45
SergeyLukjanovsdague, yup12:45
pradnyafungi:./tempest/api/compute/servers/ F821 undefined name 'attr'. if you see the test case you will find that attr is being used12:45
* fungi made a utc comment?12:45
pleia2fungi: re: requiring it for 3rd party testing12:45
*** ArxCruz has quit IRC12:46
anteayawell fungi makes a good point12:47
anteayahere is my problem, if the goal if bringing it to the ml is to achieve a reflection of community consensus, how do we know when we have that?12:47
anteayapart of the problem with the target audience is their lack of engagement with the community12:48
anteayaif I felt they would contribute to the discussion I would be all for the discussion12:48
*** dims has joined #openstack-infra12:48
fungianteaya: those who participate are those who get to be part of the consensus. those who do not, have opted out of the decision-making process12:49
anteayamy concern is that we would hinge this patch merging on a certain level of engagement from 3rd party accounts and be stuck with crickets12:49
*** CaptTofu has quit IRC12:49
*** pblaho has joined #openstack-infra12:49
anteayaif we bring it to the ml with a time frame, I am fine with that12:49
rcarrillocruzhi fungi, hopefully now is good, I added suggested change to zuul layout.yaml12:49
fungianteaya: my concern was more in getting feedback from the reviewers and developers who are having to look at these logs12:49
anteayathen once the time elapses, action can take place12:49
pleia2I'd say just give them a week12:49
sdagueso 3 requirements changes I think we should land12:50
fungito find out if it's something they're needing, rather than just something we thought sounded like a good idea12:50
pleia2+A next friday if all is well12:50
anteayaah, okay reviewers and devs, yes I can work with that12:50
fungircarrillocruz: great! i'll take a look in just a moment--thanks12:50
*** mfer has joined #openstack-infra12:50
anteayafungi: oh I needed this12:51
*** ArxCruz has joined #openstack-infra12:51
*** mwagner_wfh is now known as mwagner_wfh_bbl12:51
anteayawhen working with mikal to assess whether someone's system was fixed based on what time the latest tests ran12:51
anteayafrom a japanese server on japanese time12:51
anteayathe ryu account12:52
anteayain the middle of my night12:52
anteayait was a less than efficient conversation12:52
sdagueSergeyLukjanov: you are +2 there right? want to take a look12:52
*** ihrachys|wfh|tax is now known as ihrachys|wfh12:52
anteayabut i can take it to the ml and share my tale at the same time12:52
pleia2thanks anteaya12:52
anteayaand give it a it a week to brew12:52
anteayafungi: and regarding new projects, let me know how I can help12:53
*** hashar has joined #openstack-infra12:53
*** pblaho has quit IRC12:54
*** pblaho has joined #openstack-infra12:54
*** saju_m has quit IRC12:57
*** rfolco has joined #openstack-infra12:57
*** adalbas has joined #openstack-infra13:02
*** dprince has joined #openstack-infra13:02
*** CaptTofu has joined #openstack-infra13:03
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Add nova-specs to nova.json
*** dripton has quit IRC13:03
*** dripton has joined #openstack-infra13:03
*** salv-orlando has quit IRC13:05
*** saju_m has joined #openstack-infra13:06
*** yolanda has joined #openstack-infra13:06
SergeyLukjanovsdague, re
sdagueSergeyLukjanov: yep13:08
*** gokrokve has joined #openstack-infra13:08
SergeyLukjanovsdague, +A for and +2 for
openstackgerritA change was merged to openstack-infra/reviewstats: Add nova-specs to nova.json
*** julim has joined #openstack-infra13:12
*** dcramer_ has joined #openstack-infra13:12
*** gokrokve has quit IRC13:13
*** pradnya has quit IRC13:14
*** sdake has quit IRC13:15
*** rpodolyaka1 has joined #openstack-infra13:17
*** _nadya_ has quit IRC13:18
annegentlemorning! Can someone point me to the default stackforge doc build configuration stuff? I need to make it so that it uses a different sphinx template so pre-incubating projects don't inadvertently use the OpenStack template/logo.13:19
*** rpodolyaka1 has quit IRC13:20
*** hashar has quit IRC13:21
fungiannegentle: i don't think we have anything like that documented... is the problem that projects are using oslosphinx for their documentation templates?13:21
*** dizquierdo has quit IRC13:22
*** hashar has joined #openstack-infra13:23
sdaguefungi: probably, we probably need to hit up doug for a stylesheet that drops the logo13:24
*** nati_ueno has joined #openstack-infra13:25
*** bknudson has joined #openstack-infra13:25
fungisdague: agreed. i assume that's something which can be carried in oslosphinx or as an add-on and we don't need an entirely separate oslosphinx fork to make that happen?13:25
*** VINOD_ has quit IRC13:27
sdagueyeh, I would think it could be a flag13:28
*** jasondotstar has joined #openstack-infra13:31
*** jgallard has joined #openstack-infra13:33
*** jgrimm has quit IRC13:33
*** dansmith is now known as superdan13:34
sdaguefungi: did deletes get faster?13:35
mordreda flag or a thing we can set in the env13:36
fungisdague: deletes get faster when we're not churning as much13:38
fungisdague: the providers start ignoring or responding much more slowly to delete calls when we're building a deleting more instances13:38
annegentlefungi: pretty much it seems like that's the root cause of the problem13:38
annegentledhellmann_: fungi: anyone have ideas on how to fix?13:38
sdaguemordred: sphinx has a config file already, so something in the config seems fine13:38
fungiannegentle: i suggest a bug against oslo asking for a barebones or maybe stackforge-specific template with an option13:38
annegentlefungi: is olso.sphinx something they can opt out of?13:38
sdagueannegentle: yes13:39
fungiannegentle: it's something they opt in to by adding it to their project as a dependency and setting up their repository to support it13:39
sdagueoslo.sphinx is an optional add on to make your sphinx output take on the same format at the rest13:39
annegentlesdague: ok I'll write up an email so that projects understand what we're asking -- it happens for both docbook and sphinx built docs, where the template choice is a poor one13:39
fungiannegentle: the openstack-dev/cookiecutter repository may be a contributing factor in this, so maybe we should also have a stackforge cookiecutter fork (as i understand it, cookiecutter doesn't support multiple project templates in one repo)13:40
fungiand our current cookiecutter template is probably arranged with the assumption of official openstack projects which will default to using oslosphinx (remember it's not called oslo.sphinx any longer, for several weeks now)13:41
annegentlefungi: oh right oslosphinx13:42
mordredsdague: I mean, if it's in the env, we can enforce branding in the build env13:42
annegentlefungi: so aren't most people using the cookiecutter preincubating?13:42
mordredif it's ina config file, we have to leave it up to the project to brand itself13:42
annegentlettx: can you help with moderating through the designated sections request to the -tc list?13:42
openstackgerritA change was merged to openstack-infra/elastic-recheck: update bug to include the cleanup case
fungiannegentle: not necessarily. we have roughly equal numbers of official and stackforge projects now13:43
annegentlettx: told me too many recipients. or let me know if there's a PTL list13:43
annegentlefungi: okay13:43
ttxannegentle: done13:43
*** vhoward has joined #openstack-infra13:43
ttxannegentle: no ptl list13:43
* fungi gets current numbers, out of curiosity...13:43
anteayapleia2: ^^13:43
annegentlefungi: for docbook, we have a "built for openstack" logo template, but even so they have to sign a contrat, and currently there's no visibility into who has signed a contract so it's difficult to know13:43
annegentlettx: thanks13:44
*** alff has quit IRC13:44
fungiannegentle: makes sense. i definitely wouldn't try to tackle that issue with software alone anyway. probably better to have a stackforge-specific template or just something barebones with no logos/branding13:45
fungiwhite label, as they say13:45
annegentlefungi: my thinking is that whatever readthedocs uses is good13:45
annegentlefungi: yep13:45
sdaguemordred: sure, but couldn't we just make the docs build job fail projects that us the openstack logo but aren't in the allowed white list13:46
sdaguethen it would be on the projects to fix themselves13:46
sdagueand if people built the docs locally13:46
sdaguethey would still be right13:46
annegentlesdague: mordred: that seems fine, don't want anyone to ragequit over it (just one more hurdle might be their breaking point)13:46
mordredwell - I think what I was thinking was that there would be a normal oslosphinx that works and is unbranded everywhere13:47
sdagueannegentle: assuming an olsosphinx change where fixing it is a 1 line patch to a project13:47
mordredand that we'd inject a logo in our env13:47
sdaguesure, that's fine as well13:47
mordredthat way, third-party publication would also not be branded unless they really went out of their way13:47
fungiwell, we *might* want *some* people to ragequit. we've got quite a lot of minimally-involved stackforge projects, and no doubt quite a few abandoned ones we don't know have gone silent13:47
mordredso the 'officialness' of the stuff published on our site would be slightly more apparent13:47
mordredbut ...13:48
mordredI don't feel strongly - I think we shoudl do whatever makes the most sense13:48
annegentlefungi: heh13:48
fungicurrent gerrit project count is 130 openstack.* and 124 stackforge13:49
fungiso we're still flying just at 51% official13:50
fungifor some definition of official13:50
* fungi needs to try not to use that word since it's vague and misleading13:51
*** sandywalsh has quit IRC13:51
*** julim_ has joined #openstack-infra13:54
*** dizquierdo has joined #openstack-infra13:54
annegentlefungi: thanks for the numbers13:54
*** nati_ueno has quit IRC13:55
*** salv-orlando has joined #openstack-infra13:55
*** nati_ueno has joined #openstack-infra13:55
*** julim has quit IRC13:56
*** julim_ has quit IRC13:56
*** julim has joined #openstack-infra13:57
*** markmcclain has joined #openstack-infra13:59
*** wchrisj has joined #openstack-infra14:00
*** nati_ueno has quit IRC14:00
sdaguemordred: I disagree, we should do whatever makes the least sense14:00
fungisdague: don't we always?14:01
sdagueonly on fridays14:01
*** markmcclain1 has joined #openstack-infra14:02
*** ildikov_ has quit IRC14:02
*** markmcclain has quit IRC14:03
*** sandywalsh has joined #openstack-infra14:04
openstackgerritSean Dague proposed a change to openstack-infra/os-loganalyze: make info level not bold
*** malini_afk is now known as malini14:06
*** markmcclain1 has quit IRC14:07
sdaguefungi: that one is for mtreinish who was complaining with me cleaning up the tempest log to drop a bunch of bad debug calls it was now to hard to read a see of bold14:07
*** lukego has quit IRC14:08
*** lukego has joined #openstack-infra14:09
*** gokrokve has joined #openstack-infra14:09
*** lukego has quit IRC14:09
*** thuc_ has joined #openstack-infra14:09
*** markmcclain has joined #openstack-infra14:10
*** thuc_ has quit IRC14:11
sdaguefungi: did grizzly release?14:11
openstackgerritA change was merged to openstack-infra/os-loganalyze: make info level not bold
*** thuc_ has joined #openstack-infra14:11
fungisdague: yes, should be safe to start the great ripping14:13
fungii was going to give it until monday just to be sure there were no point release emergencies which might need correcting first14:13
* anteaya hands out party hats14:13
*** gokrokve has quit IRC14:13
*** rpodolyaka1 has joined #openstack-infra14:18
mordredsdague: ++14:20
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Authorization models added
mtreinishsdague: thanks14:21
*** rpodolyaka1 has quit IRC14:22
*** pcrews has joined #openstack-infra14:22
*** markmcclain has quit IRC14:25
mordredfungi: I cannot begin to tell you how excited I am for that14:26
*** jgrimm has joined #openstack-infra14:29
anteayamordred: grizzly release or auth models added to storyboard?14:30
*** mihgen has quit IRC14:30
*** dhellmann_ is now known as dhellmann14:33
dhellmannI'm just catching up on the backlog after a day off. Do we want a flag in oslosphinx? If so, I like the idea of setting it in the build job.14:33
dhellmannfungi: it seems from your email that the pip issue is spreading to other jobs? is the right fix there to revert the pip requirement in pbr?14:34
fungidhellmann: i don't know whether that's the "right" fix (was hoping others would chime in with reasons we might make things using pbr require newer pip than you can get from distro packages on current ubuntu lts)14:35
fungidhellmann: just that the current unfortunate consequence is that you need to pip install a newer pip now to use anything which relies on pbr14:36
openstackgerritElizabeth Krumbach Joseph proposed a change to openstack-infra/config: Move bugdaystats to to
fungibut i assume the >=1.4 is there for an actual reason14:37
fungimordred: ^ ?14:37
dstufftsystem pip is so bad :(14:38
dstufftI don't even think ubuntu LTS pip supports TLS14:38
dhellmannfungi: I don't know the history there, unfortunately. :-/14:38
*** david-lyle has joined #openstack-infra14:38
fungidstufft: probably not. it's 1.3.114:38
dstufftoh it's 1.3.1?14:39
dstufftI hought it uses 1.0 still14:39
dstufft1.3.1 does TLS14:39
dstufftstill does random scraping of shit on PyPI14:39
dstufftbut at least it's not compeltely bad then14:39
*** jnoller has joined #openstack-infra14:39
dstufftI just took a look at pbr14:39
dstufftit's setting PIP_USE_WHEEL to true14:40
dstufftwheel is only in pip 1.4+14:40
dstufftalthough it's an envvar so it wouldn't fail if pip was < 1.4, you jsut wouldn't get wheels14:40
fungidstufft: oh, actually it is 1.0 on precise. i forgot we were ending up with 1.3.1 on those machines because we had a backport of the python3-pip package from quantal14:40
fungier, from raring (13.04)14:40
dstufftfungi: there we go :V14:40
fungiwhich is using 1.3.114:40
dstufftI tried towrite a patch to add tls to 1.014:41
dstufftbut after like 3 hours I got annoyed and gave up14:41
fungii can understand that14:41
*** bnemec is now known as beekneemech14:41
*** mrodden has joined #openstack-infra14:42
*** atiwari has joined #openstack-infra14:43
dstufftfungi: yea I just grepped pbr, the only thing I can find that would need 1.4 is wheel support14:45
*** thedodd has joined #openstack-infra14:45
mordredsystem pip is silly14:47
mordredmakes no sense14:47
*** ildikov_ has joined #openstack-infra14:47
mordredif you're willing to use pip, use pip to use pip14:47
*** sdake has joined #openstack-infra14:47
*** sdake has quit IRC14:47
*** sdake has joined #openstack-infra14:47
*** krotscheck has joined #openstack-infra14:49
mordredfungi: it's 1.4 because we synced with openstack global requirements14:49
mordredwhich is only a problem if you think you're going to use a system pip14:50
mordredbut, you know, if you're doing that you're just asking for pain14:50
dhellmannmordred: do we actually need that version of pip all the time, though? and does every consumer of pbr need it?14:51
mordreddhellmann: possibly not - I could easily be convinced to downgrade the requirement14:51
mordredotoh - when installing from pip, it shoudl not be a problem14:52
*** zhiyan is now known as zhiyan_14:52
dhellmannIs it safe to have pip upgrade itself while installing an install-time dependency of something being installed with pip?14:52
*** rlandy_ has joined #openstack-infra14:52
dhellmannthe fact that I even have to ask that question makes me think no14:52
mordredhrm. nod14:52
dhellmannso it seems safer to say pbr requires pip, but to not have a specific version required14:53
*** dstanek has quit IRC14:53
dhellmannbut does pbr actually require pip?14:53
mordredpbr does not operate without pip14:53
mordredwell, I say that ...14:53
ttxjeblair: please check my facts on DepFreeze at
mordredit is possible to disable it14:53
mordredbut normal usage requires it14:53
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Another fix for
ttxjeblair: will send if you confirm sanity14:53
mordred(it doesn't require it so taht you can do things with not-pip for distro packaging purposes)14:54
dhellmannis there some way to test this without releasing an 0.8.0 of pbr with a different pip requirement?14:54
*** freyes has joined #openstack-infra14:54
*** e0ne_ has quit IRC14:54
mordredtest that it still works with 1.0?14:54
*** rlandy has quit IRC14:55
dhellmann1.0 of what?14:55
*** rlandy_ has quit IRC14:55
dhellmannI want to make it so the py33 tests for oslo don't fail so I can merge stuff.14:55
dhellmannCan I test the pbr dependency on pip for that without releasing a new pbr?14:55
mordredthey should not be failing due to pip version14:55
mordredwhat am I missing?14:55
dhellmanncontext, let me find some links14:55
mordredfungi: oh - are we using apt-based pip on the py3k nodes?14:56
mordredfungi: we should stop doing that. it's a bug in our system14:56
dhellmannmordred: oslo.rootwrap failure log:
mordredwe do not use system pip anywhere else in our infrastructure14:56
dhellmannmordred: fungi's change to strip system pip from py33 nodes:
dhellmannah, that merged14:57
mordreddhellmann: you can fix this in oslo.rootwrap14:57
dhellmannI was out yesterday, so the last I saw was:
dhellmannmordred: ?14:58
*** lcostantino has joined #openstack-infra14:58
ttxsay rootwrap three times and I appear14:58
anteayais that what it takes14:58
*** Ajaeger has joined #openstack-infra15:00
ttxhmm, looks like requirements-core could use a bit of a cleanup15:00
*** gmoro has quit IRC15:00
Ajaegerinfra team, here's on more post job fix - a single character change that I'd love to see in: Thanks!15:01
anteayattx yay15:01
* ttx generates wall of shame15:01
anteayattx you were the person to talk to about requirements-core clean up!15:01
anteayabeen bugging everybody else15:02
sdaguettx: agree15:02
* anteaya puts rootwrap * 3 aside for another day15:02
dhellmannmordred: we don't need netaddr, but otherwise that looks ok15:02
fungimordred: we were also seeing the same in git0-review's python33 unit tests, fwiw15:02
dhellmannmordred: ttx or I can fix up the patch15:02
mordreddhellmann: ok - I'm about to jump on a plane - feel free to do whatevet to it15:02
ttxsdague: let me generate good numbers15:02
dhellmannmordred: I'll handle it, thanks15:02
*** jcoufal has joined #openstack-infra15:03
Ajaegerthanks,  fungi!15:03
mordredfungi: same patch as the one above to oslo.rootwrap is applicable15:03
fungimordred: excellent--thanks!15:04
jeblairttx: that msg lgtm15:04
fungimordred: sitepackages = True shouldn't strictly be necessary though, right?15:04
ttxjeblair: coolthx15:04
mordredoh! jeez15:04
mordreddhellmann: ^^ !!!!15:05
mordreddhellmann: I copied the cantrip from nova - sorry15:05
mordredmy bad15:05
mordredfungi: THANK YOU15:05
*** saju_m has quit IRC15:05
fungimordred: also, we ripped out all the locale envvars elsewhere15:05
dhellmannso which part of this is actually fixing the problem?15:05
fungisince they were causing problems for people who didn't have those locales installed15:05
mordredk. good to know15:05
mordreddhellmann: one sec - new patch coming15:06
dhellmannmordred: ok15:06
mordreddhellmann: there you go15:06
ttxsdague: -- looks like dprince daviey gabrielhurley and joeheck did not contribute one review in last 90 days. ewindisch contributed one.15:07
fungimordred: and you had agreed previously that there was no actual reason to be overriding the user's LANG/LANGUAGE/LC_ALL in the tox.ini anyway, that it was carried over from some very early paranoia where we weren't sure whether output in other locales would break testr, if memory serves15:07
dhellmannmordred: we've had issues with skipdist and usedevelop for oslo libs in namespace packages when they use other oslo libs (not the case here, just a note)15:07
*** jcoufal has quit IRC15:07
ttxsdague: on the other hand I would formally add Sergey and mordred (they currently are in by way of the infra-core convenience), and consider dmlrr15:08
anteayattx I had thought vish was on requirements-core15:08
dstufftdhellmann: ~namespace packages~15:09
sdaguettx: I'm also a little concerned with folks that have > 90% approval rate honestly15:09
sdaguebecause I think in most cases they aren't looking that deeply into the requests15:09
* anteaya tries to remember the vish context, obviously wasn't requirements-core15:09
*** gokrokve has joined #openstack-infra15:10
mordredbtw - the stackalytics version of the russelb stats calculate "core team" by seeing if the person has a +A or +2 vote recorded in the sample period15:10
ttxsdague: agree15:10
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Coverage fix
sdaguemordred: how do they handle stable?15:11
dhellmanndstufft: afaict, the issue we've had is that we end up with 2 copies of a lib installed at the same time (one editable, one not). ISTM that non-namespace packages would have failed in less predictable ways (the lib would have worked, but may have loaded the wrong version of the code).15:11
rustlebeemordred: fine usually, except my stats count stable reviews15:11
*** VINOD_ has joined #openstack-infra15:11
*** gmoro has joined #openstack-infra15:12
ttxsdague: not formally being requirements-core, I don't think it's appropriate for me to send the cleanup email -- would you go and suggest the removals ?15:12
sdagueyeh, it will be later today15:13
dhellmannttx: I thought we all agreed to make you requirements ptl?15:13
sdagueyeh, I think we did agree to that :)15:13
sdaguewe should just do that15:13
sdaguethen you can send the email15:13
* dhellmann goes to add ttx to the group15:13
ttxhah, but then it should be added to the relmgt program :)15:13
sdaguebecause, honestly, I feel like I've been delivering too much bad news to people this week already... so would rather have someone else do it15:14
dhellmannttx: you are now in the requirements group15:14
ttxdhellmann: I suggested it originally (attach repo to relmgt program) but I think mordred shot it down15:14
ttxdhellmann: i was, by way of Release managers inclusion15:14
dhellmannttx: I don't know why it would need to change programs15:14
sdaguettx: that was mostly about ATC voting for release manager15:14
*** gokrokve has quit IRC15:14
anteayattx you don't even have to do anything, other than cull the group15:14
ttxsdague: well, we struggle in that program anyway to get enough voters15:14
anteayaI can live with that15:14
*** gokrokve has joined #openstack-infra15:15
*** CaptTofu has quit IRC15:15
ttxanteaya: btw, need to talk to you about that15:15
*** pmathews has quit IRC15:15
* anteaya stands by15:15
dhellmannttx: I  missed the change list. I can send email to the -dev list, if you can summarize.15:15
ttxanteaya: last cycle we invited PTLs to vote for the relmgr PTL15:15
*** CaptTofu has joined #openstack-infra15:15
ttxbecause the only repo under relmgt isn't that full15:15
ttxI think only Sergey and me contributed to it :)15:15
anteayasmall electorate15:16
ttxso we added stable-maint, vmt members15:16
ttxand PTLs15:16
anteayaI don't even know what repos are under relmgt15:16
ttx(stable-maint and VMT being under the program, makes sense, PTL was just an addition)15:16
* anteaya checks goveranance/projects.yaml15:16
ttxrelease-tools :)15:16
*** mgagne has joined #openstack-infra15:16
ttxanteaya: last time we didn't really have an election so we didn't need to actually apply that offer15:17
ttxanteaya: just mentioning it in case we need to have one15:17
anteayaokay if someone else wants to do what you do, I will round up some electorate for the election15:17
ttxdhellmann: it's not in any program right now. It's in the orphan group, like openstack/openstack15:18
dhellmannttx: ok, I thought it was sort of part of infra, but I'm fine with adding it to the release program if that makes things easier15:18
*** rpodolyaka1 has joined #openstack-infra15:18
ttxdhellmann: I guess it boils down to whether I'm PTL for requirements because I was personally volunteered, or because we think it belongs to Release Management15:19
ttxdhellmann: in all fairness I think I got volunteered15:19
dhellmannwell, it needs a better home than None, and I think release management makes sense15:19
dhellmannyeah, you did :-)15:19
anteayaI agree15:19
dhellmannok, I'll propose a change to the governance repo so we can talk about it at the appropriate meeting(s)15:20
anteayaboth that ttx should be ptl and that requirements makes sense in release management15:20
sdagueI think we should probably actually resolve that once and for all as a TC thing15:20
ttxdhellmann: feel free to suggest that attachment as a governance change. My last try having been denied I probably shouldn't repropose myself.15:20
*** CaptTofu has quit IRC15:20
dhellmannand then when that's decided you can propose the membership cleanup, ttx15:20
sdaguebecause I think the PTLs of the programs should be the people that vote for the release manager15:20
anteayawould increase the release management electorate as a by product15:20
sdaguenot the atc for some small repo somewhere15:20
*** rpodolyaka1 has quit IRC15:20
dhellmannsdague: I agree, though I think the stable maint team (or leader?) makes sense too15:21
*** zhiwei has left #openstack-infra15:21
ttxsdague: well, you also arguably want the people that would have to obey to that person (i.e. VMT/Stable team) to agree with that person15:21
anteayasdague: can we get that official like as a separate motion to the tc?15:21
*** packet has joined #openstack-infra15:21
anteayaI don't care who the electorate is, just that I get the correct people15:21
sdagueanteaya: sure15:21
dhellmannyeah, I didn't see moving the requirements repo under release as a way to increase the voter roles15:22
ttxsdague: "release manager" doesn't actually have any right on other projects.15:22
dhellmannso much as a way to find it a good home and leader15:22
anteayadhellmann: fair enough15:22
ttxsdague: so why would they NEED to vote ? I'm fine with them voting because we have small electorate in that team, not because I think the PTLs should all decide15:22
ttxMy power doesn't come from my title. It comes from my influence as a person. I encourage people to make the right decisions... I can't veto PTLs for example.15:23
sdaguettx: if that's the case it should be all the ATCs15:23
ttxsdague: why?15:24
ttxthe relmgt PTL is a team leader elected by team members, like any other ptl15:24
sdaguettx: I think if we have this as a designated title, it probably should have an election15:24
sdaguettx: who are those people?15:24
ttxsdague: it's no longer a title, it's a team now15:25
ttxteam members ? The release team, the VMT team and the stable maint team15:25
ttxarguably you can consider PTLs to be part of the "release management team"15:25
sdagueanyway, got to actually do some code atm :)15:25
sdagueI'm not sure I actually care that much about the eventual answer15:26
ttxsdague: heh yes, never engage in an openstack governance discussion with me :)15:26
sdagueI just think we should have it written down15:26
anteayasdague: yes15:26
sdaguebecause it's a super vague thing right now15:26
pleia2heading out to long lunch with the family15:26
sdaguenot that anyone else wants your job15:26
sdaguebut we should be explicit about how someone would take it from you :)15:26
ttxsdague: it is written @ fwiw15:26
anteayaso that should I need to send ballots to the electorate for relmgt ptl election, it is open and transparent including to me, whom I am sending ballots to15:26
ttxunder 'team members'15:27
ttxsdague: If it makes sense to add PTLs to the electorate then it should probably appear there too.15:27
* ttx will clarify15:28
sdaguesure, I guess the issue is who the team is isn't completely a function of ATC on certain repos15:28
sdagueor +2 on those repos15:28
ttxsdague: right, which is why I elaborate on "team members" there15:29
sdaguethat's the part which probably needs clarity15:29
*** pmathews has joined #openstack-infra15:29
* ttx clarifies15:29
*** david-lyle has quit IRC15:29
*** e0ne has joined #openstack-infra15:29
fungisdague: that goes potentially for other horizontal effort programs, it's just that we've sort of avoided that question elsewhere because they also tend to have repositories with some automation they use which most of their team members contribute patches to15:29
sdaguefungi: agreed15:30
sdagueI'm actually the anti pattern here, so I feel good about being the test case15:30
sdaguebecause I'm now on the release management team15:30
anteayaha ha ha15:30
sdagueand I can give good reasons why that probably makes sense15:30
sdaguebut none of them are about me being +2 on some repo15:30
sdagueso we should at least be clear about it15:30
sdagueand I think it's completely ok that it's not tied to a specific +2 role15:31
ttxEdited team members under to match what we had for last election15:32
*** jasondotstar has quit IRC15:33
ttxanteaya: ^ that should serve as electorate if need arises15:34
anteayaI just question the use of 'temporary'15:34
ttxok removed it15:35
fungiall things in the universe are temporary15:35
anteayathat is true15:35
anteayaand I take that as a given15:35
fungibut especially these sorts of things ;)15:35
anteayaotherwise noone would take the job15:35
fungiso yes, it's redundant to call it temporary, agreed15:36
anteayaso, I am happy in as much that should someone what to do what ttx does and give him a break, I know how to conduct mysefl15:36
anteayabut not spell apparently15:36
ttxanteaya: aagin, PTL of release management program != most visible member of the release team15:37
ttxI don't want to discourage people to replace me in either position15:37
*** julienvey has quit IRC15:37
sdaguettx: I expect you want to encourage it :)15:38
sdaguebecause even a week of trying to hold a few things down while you were out, I don't want that job ;)15:38
anteayaha ha ha15:38
anteayano sale there15:39
*** krotscheck has quit IRC15:39
ttxalso I want to further explore distributing the "release management" work across a team, hence me stopping usage of the "release manager" title15:40
fungiit's probably worth thinking again about how to subdivide and delegate those activities even further, as has already been done by splitting out stable release management15:40
* anteaya wonders who the most visible member of the release team is15:40
fungier, what ttx just said15:40
fungialso, not it15:40
* fungi ducks behind the crowd15:41
*** mrodden has quit IRC15:41
anteayayeah release management doesn't consist of a lot of people fighting to take the stage15:42
jeblairsdague: i imagine it was hard to do because you didn't have thierry's checklist and beer15:42
jeblairsdague: i think both are required :)15:42
ttxjeblair: the checklist is just for milestones. For FFEs I use memcache and wine15:43
ttxjeblair: sent the DepFreeze email15:43
*** jepoy has quit IRC15:44
ttxdhellmann: feel free to bounce on it to propose a requirements-core cleanup15:44
*** jepoy has joined #openstack-infra15:44
*** e0ne_ has joined #openstack-infra15:44
ttxdhellmann: 90-day reviews analysis at
*** talluri has quit IRC15:45
dhellmannttx: should we do that before or after elections? (I'm not sure if we decided it will make a difference in the voter rolls)15:45
ttxdhellmann: ideally we would do it before DepFreeze kicks in next week :)15:45
fungiJayF: if you're around, just wanted to bring your attention to my comment on (if that can be addressed, it should be able to merge today)15:46
ttxi don't want people completely out of touch to blindly ignore the freeze and +2 stuff15:46
dhellmannttx: ok, do I need a list of names to remove, or should I just propose that people reconsider their participation?15:46
* dhellmann missed that part of the discussion with sdague 15:46
ttxdhellmann: I think it's safe to remove people that have been contributing 0 reviews in the past 3 months15:47
*** jepoy_ has joined #openstack-infra15:47
dhellmannttx: makes sense15:47
ttxdhellmann: then remind the rest of the depfreeze rules I just posted, and tell them we'll clean up again soon ?15:47
*** e0ne has quit IRC15:47
*** caleb_ has joined #openstack-infra15:48
*** nati_ueno has joined #openstack-infra15:48
*** jepoy has quit IRC15:49
fungialso, the removal announcement should encourage people who *do* have an interest in checking and vetting requirements changes to start participating in reviews there15:49
fungibecause i think probably still not enough people actually do15:49
dhellmannfungi: +115:49
fungiit's one of those tragedy-of-the-commons repos15:49
*** nati_ueno has quit IRC15:51
*** rossella_s has quit IRC15:52
*** mrodden has joined #openstack-infra15:53
sdagueI also think we need to be a little careful on participation == membership15:53
*** rossella_s has joined #openstack-infra15:53
sdaguebecause there are folks that are +2ing a bunch of stuff there and not really looking that deep into things15:53
sdaguelike when sqla 0.9 went through, and turned out wasn't working at all15:54
fungii think the same goes there as for any project. people should be looking at job results and logs, checking things out for themselves, discussing relevancy, et cetera15:57
fungijust because a patch is a one-liner doesn't mean the change is a simple one15:57
*** caleb_ has quit IRC15:57
*** malini has left #openstack-infra15:57
sdagueyeh, especially with the implications here15:58
sdaguespeaking of which -  I still think we should approve15:59
sdagueI think the nova unit test issues were cleared up15:59
sdaguenow off to meet mtreinish for lunch15:59
*** Ajaeger has quit IRC16:00
*** thuc_ has quit IRC16:00
*** mgagne has quit IRC16:01
*** thuc has joined #openstack-infra16:01
fungisdague: why <=9.99 instead of <10? elsewhere in the requirements list we're well past worrying about pip versions which pull in prerelease versions without explicitly requesting them16:01
*** esker has joined #openstack-infra16:01
*** gyee has joined #openstack-infra16:01
*** caleb_ has joined #openstack-infra16:01
*** david-lyle has joined #openstack-infra16:02
*** pblaho has quit IRC16:02
fungithe <=8.99 got added that way because of a misunderstanding/mistake, from what i recall in the discussion on the ml at the time. but not having an actual interest in what openstack's declared dependencies are, i generally refrain from reviewing these except when they're emergency fixes16:03
ttxjeblair: just ran the repo/programs.yaml map and it yielded
ttxjeblair: looks like you have two missing projects under infra16:03
ttxjeblair: what shall we do with gantt ?16:03
ttxsdague: what's openstack/qa-specs ?16:04
fungittx: same as nova-specs16:04
*** mgagne has joined #openstack-infra16:04
*** yjiang has joined #openstack-infra16:04
fungittx: but for the qa program16:04
ttxsdague: ok so maybe add it to QA's section in programs.yaml16:04
fungittx: apparently multiple programs now want to do the blueprints-in-gerrit thing16:05
fungicame up while you were still vacationing16:05
*** thuc has quit IRC16:05
*** lcostantino has quit IRC16:05
*** dkliban is now known as dkliban_afk16:05
* anteaya wonders who will be ptl of the wat program16:06
*** jooools has quit IRC16:06
fungittx: and the gantt discussion similarly. it would be good to have the tc decide what does happen when an official program decides to remove itself from governance of a project (or when incubation of a project is aborted)16:06
fungittx: that would help us decide what to do with projects like gantt, melange, openstack-chef, et cetera16:07
ttxfungi: well, then it can't live under the openstack/ namespace anymore16:07
ttxfungi: original decision for melange was to move under some attic/ namespace16:07
*** bogdando has quit IRC16:07
fungittx: right, having some actual policy describing that, so that the infra team can decide on what process we follow to make that happen (such as scheduling immediate project renames or whatever)16:08
*** packet has quit IRC16:09
*** yjiang has quit IRC16:10
*** markmcclain has joined #openstack-infra16:10
*** andreaf has quit IRC16:10
fungittx: part of the concern being that "move to stackforge" isn't a great blanket solution, since stackforge isn't necessarily a designated dumping ground for failed/abandoned projects either16:11
mordredsdague: I don't think it's handling stable right now:
fungiwe could for example declare that actual abandoned projects just stay where they are and become orphans because they're no longer maintained16:11
jeblairfungi: i think our poll during the meeting ended up with most people thinking we shoud move gantt to stackforge, yeah?16:12
fungijeblair: right16:12
fungii am concerned though that we're talking about moving a project even though the interested developers plan to gut the current codebase and replace it anyway16:12
ttxjeblair: maybe pool it with the other stackforge/attic moves in the etherpad ?16:12
jeblairi think the only complicating factor, in my mind, is that it's not clear that the nova team won't want 'openstack/gantt' in the future16:13
jeblairwhich is incompatible with having openstack/gantt16:13
anteayattx do we have an etherpad of stackforge/attic moves?16:13
jeblair(we can't actually support the same project name in two namespaces; that's just an illusion)16:13
jeblair(i mean, not without lots of work)16:13
anteayattx ah the etherpad you are working on16:14
jeblairbut, i'm okay with stackforging it and just dealing with that later if it becomes an issue16:14
*** markmcclain has quit IRC16:15
JayFfungi: I saw the comment; will either remove the topic branches or create a clean fork16:17
fungii'm okay with whatever everyone else wants to do, though i still personally favor gantt being orphaned/abandoned in place and the interested developers starting a new project in stackforge under a different name16:17
fungiJayF: cool--thanks!16:17
fungisince they're not planning to keep the current codebase anyway, it didn't sound like16:17
anteayawhy does sandbox need to move to stackforge?16:18
*** krotscheck has joined #openstack-infra16:19
*** rpodolyaka1 has joined #openstack-infra16:19
*** reed has joined #openstack-infra16:19
anteayaone of the selling features, I felt, when trying to sell 3rd party systems to use it was that it was an openstack project16:19
anteayaregarding perception for newcomers16:19
fungii question that as well. it's a test repository for people to experiment with, but that doesn't mean it necessarily belongs to the stackforge namespace (similar to the "stackforge is not a dumping ground for projects nobody cares about" argument)16:20
anteayahard enough to get them to use it as it is16:20
anteayaI don't think that target audience will listen to an explaination of why they should use a stackforge repo16:21
*** rpodolyaka1 has quit IRC16:23
*** comstud is now known as bearhands16:26
*** rcleere has joined #openstack-infra16:27
*** afazekas has quit IRC16:27
*** freyes has quit IRC16:29
ttxetherpad is the new irc16:33
anteayaso that is where we can go when we net split16:34
jeblairi'm not sure how to interject there16:35
jeblairso i'll just say, yeah, i agree, we should consider deleting projects to be impossible16:35
*** rcleere has quit IRC16:35
jeblair(while technically, of course it's possible, it would be _extremely_ difficult, and we'll end up with a gerrit that returns lots of strange errors all the time16:35
*** anteaya is now known as cyan16:36
jeblairand i think it's undesirable because i don't think we should lose that history)16:36
cyanmy new nick16:36
*** talluri has joined #openstack-infra16:36
*** cyan is now known as Guest2006416:36
Guest20064I wanted to be cyan16:36
*** Guest20064 is now known as anteaya-who-was-16:37
*** anteaya-who-was- is now known as anteaya16:37
anteayaapparently I have a character limit on nicks16:37
*** rhsu has quit IRC16:37
*** gokrokve has quit IRC16:38
*** gokrokve has joined #openstack-infra16:38
*** krotscheck has quit IRC16:38
*** jogo is now known as flashgordon16:39
*** krotscheck_ has joined #openstack-infra16:39
*** JoshNang has joined #openstack-infra16:40
fungito move the conversation back into irc, so if we want to do a batch cleanup and rename/move a bunch of these all at once i'm not opposed, i would just rather not set a precedent of yet another reason to rename projects, and further increase the frequency with which we have to take our infrastructure offline for them16:42
openstackgerritA change was merged to openstack/requirements: Add pexpect to global_requirements
*** gokrokve has quit IRC16:42
JayFfungi: branches deleted16:42
fungiJayF: great! switching my -1 to a +2 then16:43
*** krotscheck_ has quit IRC16:43
JayFI have a kinda strange question: do people who have already contributed to the agent automatically become Openstack ATCs when it's merged? I think there's one or two people on our team who haven't pushed anything upstream to Ironic yet but have code already in the agent.16:44
ttxfungi: i don't think that would happen too often, and those could always wait for a planned rename window16:44
ttxfungi: as long as they get picked up one day and not forgotten (like what just happened with those)16:44
*** jlibosva has quit IRC16:44
ttxSee commit message @
fungiJayF: effectively yes, because of the way we calculate atc status based on the commit history of the projects in the program, and don't have any real way to flag only commits added on specific projects after import16:46
*** alff has joined #openstack-infra16:47
*** jasondotstar has joined #openstack-infra16:48
ttxwhere those projects should go was already discussed and decided by the TC. Now i'm fine with waiting until another important rename happens before running those16:48
fungittx: i guess that decision is official then. my fault for not noticing the commit message and petitioning the tc members to reconsider16:48
*** prad has joined #openstack-infra16:49
ttxfungi: if it's impossible/undesirable from a technical, infra perspective it's a valid objection16:49
ttxbut we are past "stackforge or not"16:49
fungittx: agreed16:49
ttxin all cases those are never "urgent". they can certainly wait until the infra is restarted for another reason16:50
anteayaoh, my fault too for not noticing it was already decided16:51
ttxso they should never add downtime16:51
fungittx: as a separate (but related) topic, should we be then considering openstack namespace projects as conveying atc status even when not part of an official program? (requirements in particular, since the current script does not consider it)16:52
*** SumitNaiksatam has quit IRC16:53
ttxfungi: if we can keep them in sync, I'd say yes16:53
fungithough i guess the governance repo might also end up with changes merged which were submitted by non-tc members, potentially people who are not otherwise atcs at all16:53
fungiopenstack/openstack not so much, since it's just a list of git submodules16:54
ttxfungi: I don't think we can use that as a source until it's cleaned up *and* kept in sync somehow, though16:54
ttxso that's probably a discussion for the future16:55
ttxnot a big deal anyway since I'm pretty sure the overlap currently is 100%16:55
fungii would rather see those official-but-orphan repositories reflected in openstack/governance:eference/programs.yaml in some way anyway16:55
ttxfungi: agreed16:55
*** sabari has joined #openstack-infra16:56
fungimakes the audit activity simpler16:56
ttxThat would make my audit script happy.16:56
* fungi nods16:56
anteayaa clean morning gate for sdague and an happy audit script for ttx16:57
anteayamorning clarkb16:59
*** VINOD_ has quit IRC17:00
*** ominakov has quit IRC17:01
*** khyati has joined #openstack-infra17:01
*** e0ne_ has quit IRC17:03
*** ildikov_ has quit IRC17:04
*** rhsu has joined #openstack-infra17:06
*** gokrokve has joined #openstack-infra17:06
*** david_lyle_ has joined #openstack-infra17:07
*** rdwrer has quit IRC17:07
*** rdwrer has joined #openstack-infra17:07
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
*** david-lyle has quit IRC17:09
*** gokrokve_ has joined #openstack-infra17:09
jeblairclarkb: just in time to see the results of what you started! ^ :)17:09
clarkbuh oh :P17:10
*** gokrokve has quit IRC17:11
jeblairoh, right pep817:11
*** caleb_ has quit IRC17:12
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
*** gokrokve has joined #openstack-infra17:12
*** jgallard has quit IRC17:13
*** gokrokve_ has quit IRC17:14
jeblairclarkb, fungi: i opened a ticket with rax on the openstackjenkins account about glance not working (it returns http 500 errors)17:15
jeblairdid that last night; no update yet17:15
clarkbjeblair: oh, I should see what I can do to get glance in hp cloud region b17:16
clarkblifeless says that is a private beta thing that we can get in on17:16
jeblairclarkb: cool, we should at least start poking at it17:16
clarkbjeblair: will try to do that after I review that nodepool change and catch up on the gantt related scrollback17:16
clarkbjeblair: yup17:16
*** markmc has quit IRC17:19
*** rpodolyaka1 has joined #openstack-infra17:19
*** adalbas has quit IRC17:22
ttxoops looks like I never hit Send on that DepFreeze email17:23
*** dizquierdo has quit IRC17:23
*** vkozhukalov has quit IRC17:23
*** SpamapS is now known as Spam17:24
annegentlettx: woo slushy!17:24
*** rpodolyaka1 has quit IRC17:24
*** adalbas has joined #openstack-infra17:24
*** Spam is now known as Maps17:24
ttxA last Glance to zuul status before calling it a week...17:25
*** melwitt has joined #openstack-infra17:25
clarkbfungi: I agree with your statement about not having another reason to restart stuff17:25
ttxoh my, it's almost empty17:25
*** rdwrer has quit IRC17:25
*** Maps is now known as CannedMeat17:25
fungittx: it's amazing what getting most of the major nondeterministic test failures will do to gating throughput17:26
*** SumitNaiksatam has joined #openstack-infra17:26
fungifailures fixed17:26
*** jnoller has quit IRC17:26
anteayafungi clarkb I just looked through the open patches for /config and don't see any create new project type patches17:26
*** thedodd has quit IRC17:26
anteayaare they there and I missed them?17:26
ttxnice peak on
ttxand the day is not over yet17:27
fungianteaya: a couple of them are in good shape to merge, the other couple we could possibly just fix for the submitters if we want17:27
ttxok, have a good weekend everyone17:27
anteayattx see ya17:28
jeblairttx: you too!17:28
fungihave fun ttx17:28
clarkbttx: thats the goal :)17:28
clarkbthough I have a chickenpox vaccine waiting for me ...17:28
clarkbjeblair: afte rreading the nodepool doc change I like this a lot17:28
clarkbI haven't read the code yet but from a config standpoint this is great :)17:29
jeblairclarkb: yaay!17:29
openstackgerritA change was merged to openstack-infra/config: Use unbound
jeblairclarkb: unfortunately, the code _looks_ like s/image/label/, but it's not.  only about half the things change.17:29
jeblairclarkb: however, i have tested this with the fake server setup and it seems to work17:30
jeblairclarkb, fungi, mordred, SergeyLukjanov: 17:32 < openstackgerrit> A change was merged to openstack-infra/config: Use unbound
clarkbdun dun dun17:31
jeblairclarkb, fungi, mordred, SergeyLukjanov: that could break absolutely everything -- nodepool images and servers alike.17:31
clarkbjeblair: on that note apparently google DNS was hijacked via BGP to do a DDoS17:31
jeblairclarkb: that's lovely!  i'm so excited we're going to use them on hpcloud!17:31
jeblairspeaking of dns...17:32
jeblairkiall: did you end up doing anything with that irc bot?17:32
*** caleb_ has joined #openstack-infra17:32
openstackgerritA change was merged to openstack-infra/zuul: Add internal noop job
*** rdwrer has joined #openstack-infra17:33
jeblairwe should restart zuul this weekend to get that ^17:33
*** caleb_` has joined #openstack-infra17:35
*** caleb_ has quit IRC17:38
*** rdwrer has quit IRC17:38
*** saju_m has joined #openstack-infra17:38
jeblairclarkb: should the username and private key vars be moved from the provider-image section to label?17:39
jeblairclarkb: i think they represent post-configuration state, and i think we expect that to be the same across all providers17:40
*** rdwrer has joined #openstack-infra17:40
jeblairbut then, it should be the same for every image, even if that image is used for multiple labels17:40
jeblairso.. maybe there's no clear fit17:40
*** NobodyCam is now known as rustler17:41
*** mkoderer has quit IRC17:41
jeblairi think maybe just leave it there for now17:41
*** rcleere has joined #openstack-infra17:43
anteayafungi: yes, looks like openstackdroid and ironic-python-agent are good to go17:43
clarkbjeblair: ya leave it for now17:43
clarkbI am good with that being image data17:43
*** ildikov_ has joined #openstack-infra17:43
anteayafungi: how would I fix the swift3 patch?17:44
anteayathe ceph one is missing an acl file17:44
* anteaya comments17:44
jeblairpuppetmaster is running its cron17:45
fungianteaya: the ceph one doesn't need an acl file. it specifies an existing acl filename in the review.projects.yaml file17:45
*** vponomaryov has left #openstack-infra17:45
*** jroll is now known as jimmyjohns17:45
* anteaya looks again17:46
fungianteaya: the swift3 patch itself doesn't need fixing, i was just hoping matt ray would either chime in that he was removing those topic branches or that there was some reason to continue development on them within gerrit17:46
anteayaline 440, so it does17:46
fungibut i couldn't find him in irc today17:46
anteayahow about the ceph one17:46
anteayaanything I can do for it?17:46
anteayait looks like it is in the same boat17:47
anteayaa noise git source17:47
fungianteaya: er, i meant the ceph one for matt ray17:48
fungithe swift3 one has a typo, in addition to having a gh-pages branch which the owner may or may not want imported into gerrit17:48
fungi(a typo in the url for the imported repo, so a bug)17:49
anteayaI could fix the typo, I guess unless we are waiting to hear back do they want the whole farm or just master17:49
fungii'm slightly less worried about deleting branches for that one than for the trove agent repo where there were more than 30, but that got cleaned up17:50
*** hashar has quit IRC17:50
fungiand same for the chef ceph one17:51
anteayamore than 30, nice work17:51
anteayai don't see a trove patch on today's menu, was that from before?17:51
anteayaa legend now17:51
fungiit only has a few which they may or may not want imported (but with names including stuff like "wip" i suspect they don't really want those)17:51
*** alff_ has joined #openstack-infra17:51
* anteaya nods17:51
fungisorry, not trove, ironic17:51
anteayaclean now17:52
*** CaptTofu has joined #openstack-infra17:52
anteayaas you said17:52
*** alff has quit IRC17:53
kialljeblair: not yet, this/next weekend :)17:54
*** derekh has quit IRC17:56
clarkbjeblair: looks like the biggest change has to do with where min_ready for each image/label is handled17:58
*** rdwrer has quit IRC17:58
*** saju_m has quit IRC17:58
jeblairclarkb: yep.  so if we want 20 devstack nodes, we don't have to set 20/7 for each jenkins master.  we just set 20.17:58
jeblairclarkb: it means we lose the ability to say "jekins01 should get 2x the nodes of the others", but we don't do that.17:58
clarkbthis is more make it scale for me friendly I like it17:59
fungii'm all for less configuring and more smart assumptions17:59
*** caleb_` has quit IRC17:59
*** rdwrer has joined #openstack-infra18:00
*** johnthetubaguy has quit IRC18:01
* SergeyLukjanov reading 1k messages scrollback18:01
fungiSergeyLukjanov: that's no way to spend your friday night18:02
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Make sure resolvconf is enabled
jeblairclarkb, fungi: ^ apparently resolvconf isn't actually enabled on some of our older servers18:02
fungioh, fun!18:02
jeblairi'm not sure why... perhaps something change in the rax image...18:02
jeblairor... and this is a worrying idea... rax maybe came around and wrote out a static /etc/resolv.conf which broke the resolvconf symlink and disabled it?18:03
fungithey do seem to do that every few months. probably a good reason to periodically rebuild our servers anyway18:03
jeblair(which they might have done as a way to update nameservers or something)18:03
fungithe image updates i mean18:03
jeblairfungi: ya18:03
jeblairi don't have strong evidence for either (we could go to backups if we really wanted to know)18:04
fungihopefully not the mass overwriting our files thing18:04
jeblairbut i did see an 'orig' file in a resolvconf directory that had different values than the current nameservers, which is what put that idea in my head18:04
fungibut if we see some revert eventually, i guess we'll know whether they do that18:04
fungioh, though in that case puppet will fix it back up for us anyway18:05
clarkbjeblair: fungi it will be so nice to have our own images :)18:05
fungiclarkb: YES18:05
sdagueso trove is now passing here -
sdagueonce we land that we can remove the trove specific job18:05
fungiclarkb: it will be nicer to have official upstream distro images (not our own) with reasonable changelogs associated with them18:05
*** jaypipes is now known as rustypipes18:05
clarkbjeblair: is resolvconf a service we can just have the service resource enable?18:05
jeblairso we have unbound running almost everywhere now, but only used random places until we merge ^ (which i think we should do asap)18:05
SlickNiksdague: just saw the conversation regarding the console log issue with the trove job.18:05
SlickNiksdague: I'll submit a change to get rid of that job once merges18:06
jeblairclarkb: it is, but it is a noop if resolvconf is configured to not update; that reconfigures it and sets up the symlink (and does an update if needed)18:06
clarkbjeblair: ok18:06
sdagueSlickNik: yep18:07
clarkbjeblair: approved18:07
jeblairsdague, SlickNik: yaaay!18:07
*** rustypipes is now known as leakypipes18:07
clarkbok back to nodepool review18:07
sdagueclarkb: or... you could +A putting trove on by default in d-g master :)18:08
sdaguebefore you run into nodepool'18:08
SlickNikthanks sdague, jeblair. :)18:09
openstackgerritA change was merged to openstack-infra/config: Make sure resolvconf is enabled
clarkbsdague: done18:09
clarkbSlickNik: welcome to the jungle18:09
*** amcrn has joined #openstack-infra18:09
SlickNikclarkb: woot! And now I have GnR stuck in my head :P18:10
*** thedodd has joined #openstack-infra18:10
*** morganfainberg_Z is now known as morganfainberg18:11
jeblairclarkb, fungi, SergeyLukjanov: in the spirit of break-everything-friday, could we merge ?18:12
jeblairclarkb, fungi, SergeyLukjanov: it might need a followup patch too after new-project-friday, but it should be harmless and that will be a nice incremental chaneg18:12
SergeyLukjanovoh, I'm ended scrolling scrollback...18:13
jeblairSergeyLukjanov: we just keep making more for you18:13
SergeyLukjanovjeblair, thank you sir!18:13
* anteaya hands leakypipes a mop and bucket18:13
clarkbjeblair: sure, I mean if we are going to break everything might as well18:13
clarkbnote I haven't even actually looked to see what change that is and I am ok with it :P18:13
SergeyLukjanovjeblair, looking on 7277918:13
clarkbSlickNik: woot that was my goal18:14
*** rdwrer has quit IRC18:14
clarkbjeblair: ooh this is a fun one18:14
jeblairclarkb: yeah, though it should still be a noop until the one after it18:14
clarkbjeblair: and we should get zuul noop in18:15
jeblairclarkb: it's merged; just needs a restart18:15
jeblairclarkb: (and then a jjb change)18:15
clarkbjeblair: is there a change up to replace the gate-noop job with noop in the layout.yaml?18:15
jeblairclarkb: not yet18:15
SergeyLukjanov++ for true noop18:16
*** jp_at_hp has quit IRC18:17
jeblairi'll write that real quick; it has a gotcha18:17
*** rdwrer has joined #openstack-infra18:17
*** dcramer_ has quit IRC18:20
*** talluri has quit IRC18:20
*** rpodolyaka1 has joined #openstack-infra18:20
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add noop job to config validator
jeblairactually, i think that's the right fix to the gotcha ^18:20
clarkbjeblair: oh nice18:20
clarkbjeblair: don't we also have the list all jobs compare against JJB jobs thing?18:21
clarkbjeblair: we might need a JJB noop job just to keep that test quiet18:21
jeblairclarkb: that's what i was on about; that function is actually in zuul18:22
jeblairclarkb: so zuul should add that to the list of jobs it checks automatically18:22
clarkboh I thought it was in shell somewhere18:22
jeblairclarkb: generating the list is in shell, but the comparison is in zuul18:22
clarkbok back to nodepool change18:23
*** rpodolyaka1 has quit IRC18:24
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Replace gate-noop with noop
*** amcrn has quit IRC18:27
adam_gdo the devstack checks that run on RAX use the standard ubuntu 12.04 image there?18:29
jeblairclarkb, fungi: hrm, so apparently that wasn't enough; we need to make /etc/resolv.conf be a symlink before that will work...18:29
jeblairclarkb, fungi: perhaps we should just go the other way, and make /etc/resolv.conf a file and ignore resolvconf?18:30
clarkbadam_g: yes, though we want to move to the pvhvm image18:30
jeblair(i don't think i realized it was quite so easy to disable)18:31
*** krotscheck has joined #openstack-infra18:31
fungiadam_g: well, they use the nonstandard "ununtu 12.04" images that rax creates and provides. we'd love to be able to use proper ubuntu-published cloud images instead (and hopefully soon we can)18:31
clarkbjeblair: and have resolv.conf point at localhost once unbound is in place as a hard set thing?18:32
jeblairclarkb: yeah, which is what we do on centos18:32
fungijeblair: agreed resolvconf hopefully shouldn't touch /etc/resolv.conf itself anyway18:32
jeblairfungi: yeah it does this:18:33
jeblair[ -L /etc/resolv.conf ] || { report_err "/etc/resolv.conf isn't a symlink, not doing anything."; exit 0; }18:33
jeblairat the top of the script18:33
fungiand debian package maintainer scripts definitely shouldn't blow away existing files/changes under /etc18:33
fungihopefully ubuntu hasn't tweaked theirs to do so18:33
*** yolanda has quit IRC18:33
clarkbjeblair: I think we can hard set it and be consistent18:34
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Disable resolvconf on ubuntu
*** jp_at_hp has joined #openstack-infra18:34
clarkbjeblair: reviewed the nodepool change. Just one thing inline18:35
jeblairclarkb: thx18:35
clarkbjeblair: and small comment on the resolv.conf change but Iam +2 there18:36
*** caleb_ has joined #openstack-infra18:36
jeblairclarkb: On centos we make it immutable; do we want to do that (can we do that? are the extended attrs enabled?) on ubuntu?18:37
clarkbjeblair: I don't think we should worry about that on ubuntu unless we need to. You can keep the exec that does it in the RedHat case and move the file out18:38
clarkbjeblair: though it may be more readable as is18:38
*** azneita has joined #openstack-infra18:38
*** thuc has joined #openstack-infra18:38
jeblairfungi: ?18:39
fungijeblair: i think consistency may be nice, but i'm also fine with it as-is until we discover that we need to make it immutable in new places18:40
*** thuc_ has joined #openstack-infra18:40
fungii'll check the ubuntu filesystem attribute situation first18:40
*** packet has joined #openstack-infra18:41
openstackgerritA change was merged to openstack-infra/zuul: Add noop job to config validator
clarkbjeblair: fungi: do we need to make it immutable for the same reason it is immutable on centos? will rax try to change it for us?18:42
*** azneita has quit IRC18:42
*** thuc has quit IRC18:43
openstackgerritA change was merged to openstack-infra/config: Revoke sudo from most jobs
fungiwe seem to be using ext3 which could complicate thingsseems to work fine...
*** e0ne has joined #openstack-infra18:43
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
fungiignore the comment about ext3, extended attributes might have been a problem, but the immutable bit has been around since at least ext2 if memory serves18:44
jeblairfungi: oh, sorry18:44
jeblairshould we immutableize it everywhere?18:45
clarkbjeblair: I am not really a fan of doing that unless necessary18:45
fungijeblair: i think consistency may be nice, but i'm also fine with it as-is until we discover that we need to make it immutable in new places18:45
lifelessclarkb: we got you in on it I thought18:45
clarkblifeless: I applied for it long ago on a different account18:45
clarkbiirc it was the ci account not the jenkins account18:45
jeblairok, let's leave it18:45
*** talluri has joined #openstack-infra18:46
clarkbright let me look up glance for the jenkins account now18:47
*** dkliban_afk is now known as dkliban18:48
*** zzelle has joined #openstack-infra18:49
openstackgerritA change was merged to openstack-infra/config: Disable resolvconf on ubuntu
*** yolanda has joined #openstack-infra18:49
*** boris-42 is now known as boris-the-blade-18:49
*** boris-the-blade- is now known as boris-the-blade18:50
*** amcrn has joined #openstack-infra18:52
*** CannedMeat is now known as SpamapS18:56
*** alff_ has quit IRC18:57
*** dkehn__ has joined #openstack-infra18:59
clarkbjeblair: fungi: while on the subject of breaking everything for friday.
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Revoke sudo from almost all jobs
clarkb77005 would be a good cleanup to have go through as right now we are in that middle spot where everything is weird19:00
clarkbjeblair: 82170 does the remaining non devstack jobs?19:01
*** caleb_ has quit IRC19:01
*** thuc_ has quit IRC19:01
jeblairclarkb: yep, should be everything19:02
*** thuc has joined #openstack-infra19:02
jeblairreview.o.o is using it's local nameserver now19:02
jeblairwhich appears to be functioning...19:02
*** mrodden has quit IRC19:03
*** mrodden has joined #openstack-infra19:04
openstackgerritA change was merged to openstack-infra/config: Switch to using script.
rustlebeecan someone add me to,members ?  (nova-docker-core)19:05
rustlebeeit's empty19:05
jeblairrustlebee: done19:05
rustlebeejeblair: great thanks19:06
*** thuc has quit IRC19:06
clarkbdoes that have a ptl group too?19:07
clarkbnope nevermind it doesn't19:07
clarkbtime for me to get lunch. I will return to breaking things after I eat19:07
clarkbjeblair: I am totally game for getting zuul noop through today and possible your nodepool changes though those are less pressing19:07
jeblairclarkb: yeah, the nodepool one is a breaking change (we'll need to stop it, merge the unwritten config change, update or drop a table, then start it)19:08
*** rpodolyaka1 has joined #openstack-infra19:08
jeblairso probably good to do during a slow period19:09
jeblairthough the multi-node stuff still needs one more change to be written19:10
*** mrmartin has joined #openstack-infra19:13
fungiif we need to drop a table when upgrading nodepool, is this also going to be a complicated update for any of the third parties who are currently cd'ing nodepool from master? (there are at least several)19:14
fungimight be worthwhile to provide a few days heads up on the -infra ml first if so19:14
openstackgerritA change was merged to openstack-infra/config: Add a script to manage IRC perms
SergeyLukjanovjeblair, this one too please
SergeyLukjanovfungi, heh, for sahara-ci we'll just need to drop our 6 nodes :)19:17
SergeyLukjanovfungi, ++ for ml heads up19:17
*** weshay has quit IRC19:18
*** salv-orlando has quit IRC19:20
jeblairyeah, i'll send out an announcement, but we've never made any promises about nodepool master.  it has no schema migration framework or even any tests, and it has '' still hardcoded into it all over the place19:20
*** yolanda has quit IRC19:21
jeblairso i don't think we should delay for that reason19:21
fungifair enough19:21
jeblairbut i'll send the mail now19:22
openstackgerritA change was merged to openstack-infra/config: Add #openstack-sahara channel to accessbot
*** reed has quit IRC19:23
*** e0ne has quit IRC19:24
*** e0ne has joined #openstack-infra19:25
*** CaptTofu has quit IRC19:28
*** mrodden1 has joined #openstack-infra19:28
*** CaptTofu has joined #openstack-infra19:29
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Create propose_translation_update for horizon
*** mrodden has quit IRC19:31
*** yolanda has joined #openstack-infra19:32
*** krotscheck_ has joined #openstack-infra19:33
*** CaptTofu has quit IRC19:33
*** krotscheck_ has quit IRC19:34
*** krotscheck has quit IRC19:34
*** khyati has quit IRC19:34
*** krotscheck_ has joined #openstack-infra19:34
*** krotscheck_ is now known as krotscheck19:34
*** thuc has joined #openstack-infra19:35
*** adam_g has left #openstack-infra19:38
*** adam_g has joined #openstack-infra19:38
*** amotoki has quit IRC19:38
*** dcramer_ has joined #openstack-infra19:38
*** krotscheck has quit IRC19:39
*** yolanda has quit IRC19:41
*** rpodolyaka1 has quit IRC19:42
openstackgerritSergey Lukjanov proposed a change to openstack-infra/devstack-gate: Don't enable savanna service for sahara
*** hashar has joined #openstack-infra19:45
openstackgerritA change was merged to openstack-infra/storyboard: Coverage fix
*** krotscheck has joined #openstack-infra19:48
clarkbok back from lunch19:50
*** krotscheck has quit IRC19:51
*** krotscheck has joined #openstack-infra19:51
*** jp_at_hp has quit IRC19:52
adam_gwhich instance flavor is used on the RAX cloud?  trying to find out why this works for me on RAX but fails in the gate:
*** thuc_ has joined #openstack-infra19:53
*** rfolco has quit IRC19:53
clarkbjeblair: see comment on
clarkbadam_g: you can see them at
clarkbadam_g: base-image: 'Ubuntu 12.04 LTS (Precise Pangolin)' being one example (I think that is the precise one being used19:55
adam_gclarkb, oh cool. those are the details i was looking for19:55
adam_gevery job gets a fresh instance, correct?19:55
clarkbadam_g: we slap some stuff on top of that image and take a snapshot then boot that so it isn't quite 1:1 but should be close19:55
clarkbadam_g: yup every devstack job happens on a one time use slave19:56
adam_gclarkb, interesting. ill see if running from a snapshot changes anything19:56
*** thuc has quit IRC19:56
*** thuc_ has quit IRC19:57
adam_gfungi, that is what gets slapped on top?19:57
fungiadam_g: that and the other scripts it calls from the same directory should give you an idea of how the machines end up configured on top of the base image19:57
clarkbfungi: jeblair: where are we on new projects?19:57
adam_gfungi, thanks19:57
fungiclarkb: there are four, several of which need reviews or possibly fixes applied19:58
*** rpodolyaka1 has joined #openstack-infra19:58
fungiadam_g: basically nodepool boots an instance built from the base image clarkb mentioned, then scps the nodepool scripts onto it and runs that one (which calls some of the others), then shuts it down, takes a snapshot and boots jenkins slaves from them. makes new images that way nightly19:59
*** SumitNaiksatam has quit IRC20:01
clarkbfungi: should I review and approve those that don't need work and push patches to the others?20:02
clarkbfungi: then we can start creating them?20:02
jeblairclarkb: back, and responded to
*** krotscheck has quit IRC20:02
adam_gfungi, cool. easy enough to reproduce20:03
JayFfor, devananda asked if we should change the group for tags to the ironic-ptl or ironic-milestone group but to check with you guys first -- I'm assuming that using ironic-milestone should be fine?20:03
*** krotscheck has joined #openstack-infra20:03
*** Ryan_Lane has quit IRC20:03
clarkbJayF: we use ptl for tags most places20:03
fungiJayF: tags usually use the ptl group since it should be release managers/delegates taking care of pushing signed tags on projects20:04
ArxCruzALL: Is there a way to, using o-gate to define my own Cirros image?20:04
JayFclarkb: is there an ironic-ptl group? I didn't see one referenced anywhere in the configs that I was looking through.20:04
fungithe ptl group can be multiple people delegated by the ptl for those duties20:04
clarkbJayF: if the group doesn't exist it will be created20:04
*** jgrimm has quit IRC20:04
JayFokay, I'll use ironic-ptl then for that case, would that be fine with all of you? and if it needs creation, we should make sure devananda is the first member20:05
jeblairJayF: wfm20:05
*** saju_m has joined #openstack-infra20:05
devanandathanks guys20:05
clarkbjeblair: fungi: you ok with me approving those new-project changes? some of them will pull in more gate-noop job instances20:05
jeblairclarkb: yeah, that's secondary20:05
*** khyati_ has joined #openstack-infra20:05
*** krotscheck has quit IRC20:05
jeblairi marked the ironic agent one wip for JayF's change20:06
*** krotscheck has joined #openstack-infra20:06
fungiclarkb: sure20:06
fungishould be safe20:06
clarkbI approved openstackdroid. Will wait for JayF to update the ironic agent20:06
jeblairis someone going to fix the ceph one?20:06
clarkband I will push patches to the other two to fix them20:06
clarkbjeblair: I can do it20:07
jeblairclarkb: cool, thx20:07
*** rfolco has joined #openstack-infra20:07
*** rpodolyaka1 has quit IRC20:07
*** mihgen has joined #openstack-infra20:07
fungiclarkb: well, one of them needs a minor typo fix and possibly some branches deleted, the other just possibly needs branches deleted20:07
openstackgerritA change was merged to openstack-infra/config: Initial openstackdroid commit for Stackforge, with review changes
openstackgerritJay Faulkner proposed a change to openstack-infra/config: Add openstack/ironic-python-agent
fungiso if we're cool with doing that after import, then i'll switch my vote to +2 on it20:07
jeblairif it's just a gh-pages branch, i vote we just merge as is and delete later20:08
clarkbfungi: jeblair ya actually I think that is easier than me forking on github and cleaning it up there20:08
fungithe gh-pages branch is on the one with the url mistyped, so that's safe20:08
fungionce corrected20:08
clarkbyup I will correct the url20:08
clarkband the ceph cookbooks repo has the same problem20:08
JayFUpdated ironic-python-agent -- I think this should be g2g now20:08
fungithe other one is matt ray's and i haven't been able to find him on irc to get feedback20:09
clarkbI can clean up the branches in gerrit if that is what they want20:09
clarkband we can import it as is20:09
fungiyup, okay20:09
*** krotscheck has quit IRC20:09
*** krotscheck has joined #openstack-infra20:09
*** rpodolyaka1 has joined #openstack-infra20:10
jeblairlists.o.o is using its unbound server20:10
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add swift3 into stackforge
clarkbfungi: ^ how does that look?20:10
openstackgerritA change was merged to openstack-infra/config: Another fix for
clarkbjeblair: lists and review were the two problematic hosts right?20:10
jeblairclarkb: yep20:11
jeblairclarkb: and, apparently from the several tickets on the openstackjenkins account, the jenkins slaves actually have been a problem for them20:11
fungiclarkb: lgtm20:11
jeblairfurther evidence those dns errors may be caused by their rate limiting20:12
openstackgerritA change was merged to openstack-infra/config: Add openstack/ironic-python-agent
fungijeblair: clarkb: and ask.o.o, but we don't control that one20:12
jeblairclarkb, fungi: did you see the disappointing response from hpcloud on dns?20:12
JayFtyvm guys for getting the agent in20:12
dhellmanngerrit question: If I have a person's name and email, is there some way to get the gerrit id that I need to put in the json config file for reviewstatus? It looks like it wants a short-form of their id or a nickname or something.20:12
clarkbjeblair: not yet /me looks20:12
fungijeblair: yeah, at least it's under our "control" (to pick some resolvers somewhere at random?)20:13
jeblairJayF: since our automation is broken, it's probably another hour or so until the project is actually imported20:13
clarkbjeblair: that seems like a canned we don't quite understand the question response20:13
jeblairclarkb: yep.20:13
*** krotscheck has quit IRC20:13
*** rfolco has quit IRC20:13
*** mallow123 has joined #openstack-infra20:13
clarkbjeblair: also if the 206 address was specified on the subnet we didn't specify it20:13
clarkbI didn't at least20:13
fungijeblair: one speculation on the jenkins slave lookup failures was that we were hitting previously used ip addresses which had been blacklisted in their resolvers and that they don't necessarily lift their blacklists immediately when addresses get recycled20:13
*** afazekas has joined #openstack-infra20:14
clarkbfungi: oh interesting20:14
jeblairfungi: possible.  it's also possible we were the previous owner of that ip address.  :)20:14
*** rpodolyaka1 has quit IRC20:14
fungiextremely possible ;)20:14
openstackgerritA change was merged to openstack-infra/config: Add Ceph support to existing Chef cookbooks
jeblairfungi: but that does help to explain the occurences when it seemed like it happened very early20:14
openstackgerritA change was merged to openstack-infra/devstack-gate: trove is an integrated project add it by default
fungiright, at times where we shouldn't (yet) have done much in the way of lookups20:15
jeblairwe should do a devstack-gate run where we tcpdump port 5320:15
fungijeblair: another possible mitigation there... have the nodepool ssh acceptance test try to do a dns lookup before marking the node ready?20:15
jeblairfungi: we could.  if we still see the problem in our forwarding configuration, we might consider that (or running our own long-lived resolvers for forwarding, or just doing recursive on the slaves)20:17
clarkbjeblair: which seems to be what hpcloud would like us to do20:17
jeblairi'm going to kick off image builds20:17
openstackgerritA change was merged to openstack-infra/config: Add swift3 into stackforge
clarkbonce ^ makes it onto the git replicas we are good to run manage projects right?20:17
jeblairclarkb: let's make sure the right people really understand the question/issue before we assume that's the final word20:17
jeblairclarkb: because i agree it seemed like a canned response20:18
jeblairclarkb: (also, neither of our provider images make it easy to change dns!)20:18
clarkbwell hp is saying we should change it via the neutron dhcp server20:18
clarkbwhich isn't terrible if we can do that in rax land too20:18
clarkbwhen they neutron20:18
*** dprince has quit IRC20:19
jeblairclarkb: oh is that what they are saying20:19
jeblairthat's slightly better.  but still kind of copping out on the "hey why is your dns breaked?" question.20:19
*** mwagner_wfh_bbl has quit IRC20:19
clarkbjeblair: yup20:19
*** mwagner_lap has joined #openstack-infra20:19
fungii got that, but it still doesn't address why they differed, nor whether they're willing to fix dnssec on theirs20:19
*** mwagner_ has joined #openstack-infra20:21
*** jasondotstar has quit IRC20:21
*** vkozhukalov has joined #openstack-infra20:21
*** SumitNaiksatam has joined #openstack-infra20:21
jeblairreally need a way to trigger image updates for everything in nodepool :/20:22
*** openstackgerrit has quit IRC20:22
*** rossella_s has quit IRC20:22
*** openstackgerrit has joined #openstack-infra20:23
*** harlowja_still_a has joined #openstack-infra20:23
clarkbnodepool image-build-all-the-things20:23
*** krotscheck has joined #openstack-infra20:24
*** pcm_ has joined #openstack-infra20:25
jeblairapparently we have 33 images20:25
pcm_Can I get core review of please?20:25
*** rfolco has joined #openstack-infra20:27
*** krotscheck_ has joined #openstack-infra20:27
clarkbpcm_: we try to let the proper core bits of openstack/requirements review those unless it is an emergency. I think there has been talk about reworking how core works for that though20:27
clarkbsdague: ^ is that something you wanted to have done?20:28
fungipcm_: the requirements core reviewers are those listed as direct members of,members20:28
*** krotscheck_ has quit IRC20:28
jeblairokay, image rebuilds running in 33 screen windows20:29
pcm_fungi: clarkb: Thanks, so I need to contact them specifically? (been waiting a long time for review)20:29
clarkbpcm_: ideally yes. Infra tends to only touch that if requirements are breaking infra20:30
pcm_gotcha. Thanks!20:30
sdaguehonestly, not at this stage of the release20:30
sdaguewe need to stop using a million different test libraries20:30
*** esker has quit IRC20:31
pcm_sdague: It's specifically for HTTP REST testing.20:31
sdaguepcm_: sure20:31
sdaguebut we also have webtest, wsgi_intercept, and a host of other things20:32
sdagueand we're basically coming up on dep freeze for release, so right now isn't the time to add new things20:32
pcm_sdague: The alternative is that I essentially pull that source code into the test code base, or write a similar mechanism.20:32
sdaguepcm_: where is the dependent test that's calling this?20:33
pcm_I have it labeled notest... so it doesn't run... but have been using it in my environment for weeks. The review has been out for weeks as well.20:34
*** jasondotstar has joined #openstack-infra20:35
*** saju_m has quit IRC20:35
dtroyerpcm_: fwiw, we already have a client dependency on httpretty for this sort of testing.  I know they're not exactly the same thing, and for most I prefer not monkeypatching the runtime, but that's already in...20:35
sdaguepcm_: so, no, not at this point in the cycle. Also, my general understanding is this "start the whole wsgi stack" approach to testing neutron is supposed to get rethunk in juno20:36
clarkbfungi: git01 and git05 have swift3 at this point but not 02-0420:36
clarkbfungi: will ping when I think those repos are properly in place20:36
pcm_sdague: i'd encourage you to look at the library, as it is NOT starting a whole stack.20:36
dtroyersdague: it's actually a less-invasive and requests-specific parallel to httprettty20:37
dtroyerI wish we'd seen it about six months ago20:37
sdaguedtroyer: that's fine, but at this point in the cycle I don't want to keep piling on new requirements. We're weeks into freeze20:37
jeblairclarkb, fungi: qq: i added a "dig" command to sanity check the dns configuration in the bootstrap script.  it looks like at least one centos image doesn't have that installed.  should i make bind-utils be installed everywhere or remove the sanity check?20:38
sdagueso I think it's a juno topic20:38
*** rpodolyaka1 has joined #openstack-infra20:38
sdagueand I think in doing so we do need to have a conversation about test library convergence20:38
dtroyersdague: totally agreed…  I'm more just slighty sad out loud having fought this exact problem most of Teusday afternoon and resorting to httpretty since it's already in keystoneclient20:38
fungisorry, had to step away--catching back up now20:39
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Be diligent in unmounting /proc/xen for Ironic
sdaguepcm_: so one we've cut stable/icehouse on requirements please start a mailing list thread on it20:39
sdaguebecause if this is a better version of httpretty, than I'd like to get httpretty removed as part of it20:39
*** thomasem has quit IRC20:40
pcm_sdague: cool! I really like the library. Thanks. Will do.20:40
clarkbfungi: swift3 is everywhere now as are the other 3 new projects20:40
clarkbfungi: I think that means we are ready to run manage-projects right?20:40
pcm_sdague: Should we do anything on the code review (dispositioning wise)?20:40
*** jasondotstar has quit IRC20:41
*** mrmartin has quit IRC20:41
sdaguepcm_: I just put a -2 on it for now which I'll remove once we open up the trees again20:41
fungijeblair: installing dig everywhere seems like a good idea to me--we did the same for other useful investigative tools like tcpdump and netcat20:41
clarkbhave we done the first RCs yet? that must happen soon if not20:41
sdagueI'd say start the email thread now, but I honestly expect it will get ignored as people work on the releases20:41
clarkbsdague: Iask about RCs because after the ton of bug fixes this week seems like RCs would be a good thing20:41
sdagueclarkb: no rcs yet20:41
dtroyerpcm_: FWIW,  jamielennox is going to be very interested in this topic, he introduced httpretty…20:41
pcm_sdague: thanks. How will I know that reuqirements has been cut? I work on neutron, so not sure how tell.20:41
sdaguepcm_: it will be once there are stable/* trees for all projects20:42
fungiclarkb: should be fine to run manage-projects now. please run it with the -d option and tee/redirect to a file so we can figure out what breaks (if anything)20:42
sdaguewhich is probably 2 - 3 weeks, I forget when in the cycle we go from mp -> stable20:43
*** prad has quit IRC20:43
pcm_sdague: OK. I'll wait and then raise on the ML. Would love to see discussion on it and hopefully acceptance.20:43
*** rpodolyaka1 has quit IRC20:44
sdagueyeh, if dtroyer's assessment is accurate, I think it shouldn't be an issue.20:44
sdagueit's just a timing problem right now20:44
pcm_sdague: sure np. Appreciate the attention to this.20:44
clarkbfungi: ok doing that now20:44
*** vhoward has left #openstack-infra20:45
clarkbfungi: jeblair it is running in a root screen and is dumping logs to a file via tee20:46
sdaguespeaking of requirements - dhellmann you around?20:47
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Make sure dig is installed everywhere
jeblairclarkb, fungi: brown bag fix pls ^20:47
clarkbjeblair: +2'd20:48
jeblair(that is installed on hpcloud but not rax, btw)20:48
clarkbfungi: feel free to approve20:48
sdaguejeblair: "tests and radical idea like schema migrations" :)20:49
clarkbdevananda: you have been added to the ironic-ptl group20:52
openstackgerritA change was merged to openstack-infra/config: Make sure dig is installed everywhere
clarkbdevananda: and the agent's core group. Freel free to add members to that core group20:52
clarkbI suppose I can add JayF too20:53
devanandaclarkb: thanks. is there a substantive difference between -ptl and -milestone groups?20:53
clarkbdevananda: ptl group is for ptl tasks eg doing releases20:53
clarkbmilestone is for other ttx related things iirc20:53
devanandaah, gotcha20:53
fungidevananda: -milestone groups get to pick what's approved to the milestone-proposed backport branch during release freeze20:53
devanandafungi: ahhh. does that also apply post-release, eg. for who gets to pick backports?20:54
clarkbJayF: what is your gerrit account number? you can find it on the gerrit settings page20:54
fungidevananda: that said, right now openstack/python-ironicclient can get tags pushed by any members of ironic-core, which probably does merit solving20:54
JayFclarkb: 1034220:54
devanandafungi: indeed20:54
clarkbJayF: you are now ironic agent core too20:54
*** prad has joined #openstack-infra20:55
devanandathanks guys!20:55
*** prad has quit IRC20:55
*** SpamapS has quit IRC20:58
jimmyjohnsb 1020:59
*** julim has quit IRC20:59
*** SpamapS has joined #openstack-infra20:59
*** SpamapS has quit IRC20:59
*** SpamapS has joined #openstack-infra20:59
*** pdmars has quit IRC21:00
dhellmannsdague: here21:00
*** mfer has quit IRC21:00 Failed to clone stackforge/openstackdroid21:00
jeblairclarkb: lol building images on new project friday :)21:00
clarkbjeblair: whoops21:01
clarkbjeblair: manage-projects is done now so you should be clear to try again >_>21:01
*** thuc has joined #openstack-infra21:02
clarkbI have seeded the initial group members for all projects except for the ceph one as that is piggy backing on existing chef cookbook groups21:03
*** dkehn__ has quit IRC21:03
*** dkehn_ has quit IRC21:04
sdaguedhellmann: as req core, you want to provide a +2 on this one -
clarkbI don't see anything out of place with the new projects21:05
clarkbI am pretty confident in manage-projects now21:05
sdaguedhellmann: it's one of the last 2 req changes I think we need to land pre release21:05
dhellmannsdague: we've tested that? it's so late in the cycle...21:05
sdagueit's self testing21:05
*** dkehn_ has joined #openstack-infra21:05
dhellmannsdague: famous last words? :-)21:06
sdague0.9 is used on all those test runs21:06
sdagueyou can look for yourself21:06
dhellmannsdague: +2 a21:06
dhellmannsdague: no, I just meant we've seen things pass tests before and then introduce instability21:06
sdaguealso note markmc's comments - all the distros are going to use it anyway21:06
sdaguewe're not helping them by not running it upstream21:06
clarkbhuh did I tee manage-projects wrong? the log file is zero bytes21:07
clarkbfungi: does that go to stderr instead of stdout?21:07
dhellmannsdague: yeah21:07
*** hashar has quit IRC21:07
clarkbwell good thing it looks like everything is happy21:07
dhellmannsdague: you're a gerrit guru, maybe you can answer this: I want to put together a list of gerrit ids for the cores in the requirements review group, but all I have are names and email addresses. How do I get the id value I need for the reviewstatus app rustlebee wrote?21:07
fungiclarkb: oh, yeah i usually do 2>&1 | tee foofile21:08
clarkbdhellmann: there is a hack to get that number. click on the persons name in gerrit and the url it goes to ends in that number21:08
fungiclarkb: sorry 'bout that21:08
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Correct invocation order of install_devstack_dependencies
clarkbdhellmann: not sure if there is a programmatic way to get it that doesn't suck21:08
clarkbfungi: its ok, I now know :)21:08
dhellmannclarkb: the other example files have usernames not numbers, does the tool work with numbers?21:08
clarkboh it wants usernames why did I read that as account numbers21:09
jeblairclarkb, fungi: is urgent21:09
jeblairdhellmann: usernames show up in gerrit queries over ssh21:09
jeblairdhellmann: (i wrote that patch actually)21:09
sdaguedhellmann: mostly just gets done by knowing those things... I've never reversed it out beyond that21:09
jeblairdhellmann: so if you run "ssh review gerrit query 82193"21:10
clarkbfungi: in any case manage-projects seems to have gone smoothly. I am pretty confident that the only remaining problem is the orchestration across nodes21:10
jeblairdhellmann: you can see my username as the owner21:10
dhellmannjeblair: is there a way to dump a group list over ssh?21:10
jeblairdhellmann: so if you run a similar query for changes owned or reviewed by the people you're interested in, you can get them21:10
* dhellmann should read some gerrit docs21:10
clarkbnot with our old gerrit21:10
dhellmannjeblair: ah, ok21:10
clarkbmikal got it added to new gerrit though21:10
clarkbso around end of april you should be able to do it21:11
openstackgerritA change was merged to openstack-infra/config: Correct invocation order of install_devstack_dependencies
*** alff_ has joined #openstack-infra21:17
jeblair3 new images are ready (a bare-precise and 2 bare centos6) all in rax21:18
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Comments UI
*** rcleere has quit IRC21:22
*** mbacchi has quit IRC21:22
*** nicedice has joined #openstack-infra21:24
clarkbjeblair: when you are done >_>21:25
*** talluri has quit IRC21:26
*** talluri has joined #openstack-infra21:26
*** afazekas has quit IRC21:26
*** sandywalsh has quit IRC21:27
jeblairdone.  but it won't get built anyway since tripleo thinks we're over quota21:27
*** e0ne has quit IRC21:27
clarkboh right21:29
*** jepoy_ has quit IRC21:29
clarkbwas lifeless leaving that as in order to debug?21:29
*** jepoy has joined #openstack-infra21:29
clarkblifeless: ^ I saw the bug you submitted to nova about it but that isn't tacking the state of broken in your cloud21:29
clarkboh right lifeless is weekending21:29
*** dstanek_afk has joined #openstack-infra21:31
*** alff_ has quit IRC21:31
jeblair8/33 images21:33
jeblair(still only unit test nodes)21:33
*** jepoy has quit IRC21:33
clarkbguys I am braving the hipchats for glance access21:34
jeblairclarkb: good luck!21:35
*** jamielennox|away is now known as jamielennox21:38
clarkbjeblair: I am told the request is being submitted21:39
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Be diligent in unmounting /proc/xen for Ironic
*** rfolco has quit IRC21:39
*** rpodolyaka1 has joined #openstack-infra21:40
fungii look forward to glancing all the things21:41
fungigotta run to dinner though--will catch up in an hour or so21:41
*** dstanek_afk is now known as dstanek21:42
jeblairfungi: enjoy!21:42
jeblairclarkb: cool.  if i don't see a response on the rax ticket next week i'll send emails21:42
*** thedodd has quit IRC21:43
*** rfolco has joined #openstack-infra21:44
*** rpodolyaka1 has quit IRC21:45
*** torgomatic has joined #openstack-infra21:46
clarkbjeblair: fungi: the access I am requesting will make the images private to our project which I figured is a good first step21:46
*** _nadya_ has joined #openstack-infra21:47
clarkbthere may be ways to make them more public if we want that at some point but that is apparently more access and trickier21:47
clarkbso I figured we can worry about that later once we figure out what we should do with these images uploads21:47
*** jeckersb is now known as jeckersb_gone21:49
*** packet has quit IRC21:49
*** mrodden1 is now known as mrodden21:50
jeblairclarkb: yeah, and we can just drop these on static.o.o to serve to people21:50
jeblairclarkb: though we may want to run a public glance too, just so we're that cool.21:51
marunwhat exactly does DEVSTACK_GATE_NEUTRON configure?21:52
sdaguemarun: it turns on the neutron services and doesn't turn on n-net21:52
marunsdague: perfect, thanks!21:52
marunjeblair: ah, cool.21:52
marunjeblair: my grepping foo is weak :/21:53
maruni was only looking in the config repo21:53
jeblairmarun: and several points in for the gory details21:53
jeblair(depending on branch)21:53
marunjeblair: good to know where to look, thanks.21:54
jeblairmarun: np21:54
openstackgerritDoug Hellmann proposed a change to openstack-infra/reviewstats: Add requirements project config file
jeblairINFO:nodepool.ImageUpdater:Image in rax-dfw is ready21:55
*** pcm_ has quit IRC21:55
jeblairthats the first new devstack image ^21:55
pleia2oh, I didn't even notice, jenkins decided to ignore
pleia2no verified21:56
clarkbpleia2: recheck no bug it21:56
clarkbthe zuul machinery is looking for a -1 or +1 check vote but there is no vote21:56
clarkbwe can force one with a recheck no bug21:56
pleia2gotcha, thanks21:56
pleia2there it goes21:57
*** jgrimm has joined #openstack-infra21:58
*** e0ne has joined #openstack-infra21:58
openstackgerritMaru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing
*** e0ne has quit IRC22:00
*** torgomatic has left #openstack-infra22:00
openstackgerritMaru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing
*** e0ne has joined #openstack-infra22:00
clarkbmarun: out of curiousity why are those tests not in tempest directly?22:00
marunclarkb: because tempest doesn't want them22:00
clarkbthat'll do it22:00
marunclarkb: Plus, I'm trying to be clever.22:01
openstackgerritA change was merged to openstack-infra/config: Setup the tripleo-test-cloud as a provider for Fedora instances
marunclarkb: I'm writing functional tests that directly target neutron plugins through an extraction layer.22:01
marunoops 'abstraction layer'22:01
*** _nadya_ has quit IRC22:01
marunclarkb: and then the same tests can use a different abstraction layer to target deployed neutron via the tempest rest client22:01
marunclarkb: birds, stones, etc22:02
marunsdague: reviews welcome  :)22:03
*** dstanek has quit IRC22:04
marunclarkb: hmm, I guess I can't use bash variables like that?22:05
*** e0ne has quit IRC22:05
openstackgerritMaru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing
clarkbmarun: no you can't I commented on how to fix it22:05
clarkbyou can also just go with $CMD_PREFIX22:06
*** jepoy has joined #openstack-infra22:06
marunah, I just reverted to copy/paste22:06
marunis that ok?22:06
clarkbcopy paste is fine too22:06
jeblairclarkb: btw, i think with the nodepool label change, we can build 'check' nodes that aren't based on different images22:06
marunclarkb: great, fixed.22:07
clarkbjeblair: oh man22:07
jeblairclarkb: so testing a provider/image combo will be easier22:07
clarkbjeblair: right we can just add a label and use the existing image when we are happy with a thing22:07
sdaguemarun: so is the only reason you are pulling in tempest is for our rest client?22:08
sdagueyou are going to need to do a lot more configuring of tempest than a pip install22:09
sdagueto make it run22:09
*** bknudson has quit IRC22:09
marunsdague: that's devstack's job22:13
marunsdague: or am I preventing that from happening… hmmmm22:13
marunsdague: is there to ensure that tempest gets configure but doesn't run?22:14
*** dstanek has joined #openstack-infra22:14
sdaguehmm... maybe this works22:14
sdaguethe pip install is weird though22:14
marunsdague: yeah, I don't know a better way to do it22:14
flashgordondumb question: why can't I hit
sdagueflashgordon: because it's been gzed22:15
marunsdague: we'll want to add an explicit dependency but I'm hoping to delay doing so until tempest libification is complete22:15
flashgordonsdague: doesn't work either22:15
marunsdague: so a manual install in the tox venv was the best I could come up with22:15
marunsdague: do you have an alternative suggestion?22:15
flashgordonneither does
sdaguemarun: nope22:15
sdaguemarun: who thinks they are working on the libification. That we should definitely have a qa-spec proposed for22:16
flashgordonclarkb: ^22:16
marunsdague: Maybe I'm just extrapolating in my head on some other effort22:16
clarkbflashgordon: files didn't upload then22:17
clarkbflashgordon: where is the chagne you are seeing that for?22:17
marunsdague: jaypipes has mentioned libification more than a couple of times to me but maybe that doesn't encompass the rest client and its configuration.22:17
sdaguebecause there are lots of demands on tempest, especially with defcore coming up, so I really don't want people to think that's something that's emminent or even a priority22:17
clarkbflashgordon: derp22:17
clarkbits 4505722:17
uvirtbotLaunchpad bug 1221922 in nova "test_change_password_for_alt_account_fails timed out in swift gate job" [Undecided,Incomplete]22:17
clarkbflashgordon: we only keep 6 months of logs22:17
clarkbflashgordon: I think september was more than that ago22:18
flashgordonclarkb: ahh sept 2013 yeah22:18
flashgordonis it possible to get 1 year of logs?22:18
clarkbflashgordon: not without twice as much disk22:18
marunsdague: apologies, don't mean to imply its necessary for this job, just that it was one direction things could go.  do you want me to change the comment in the job?22:18
clarkb/dev/mapper/main-logs          11T   11T  711G  94% /srv/static/logs22:18
clarkbjeblair: fungi ^ we are getting close to our limits there22:19
marunsdague: the alternative would be to add an explicit test dependency in neutron, I'd like to prove the concept before I propose that to neutron folk22:19
clarkbflashgordon: so if you have 22TB of disk laying around for 22TB of compressed logs sure22:19
marunsdague: explicit dependency -> on tempest22:19
flashgordonmordred: can I get more disk please?22:19
flashgordonclarkb: 22TB whoa22:19
clarkbsdague: flashgordon I definitely think that has been a large part of recent testing22:20
clarkbsince it was 8.5TB last I looked not long ago22:20
flashgordonahh makes sense22:20
clarkbflashgordon: yes it is insane22:20
sdaguemarun: so I'm totally cool with you doing things as this, just realize the tempest interfaces are fluid, so it will break22:20
marunsdague: fair enough, it'll probably stay non-voting for the immediate future anyway.22:20
sdagueit will be good data22:21
sdagueto try out the idea22:21
*** dhellmann is now known as dhellmann_22:21
marunsdague: I'll update the comment for clarity at least, then feel free to +2 ;)22:21
sdaguewell, I'm only a +1 there22:21
sdaguebut I did at that22:21
clarkbwe are due for that number to fall slightly looks like so I don't think this is an emergency prior to the weekend22:21
openstackgerritMaru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing
clarkbflashgordon: this is why we want to get logs into swift22:22
clarkbflashgordon: apparently we have more terabytes in swift22:22
marunsdague: danke22:22
*** gokrokve has quit IRC22:24
flashgordonclarkb: :)22:24
openstackgerritA change was merged to openstack/requirements: allow sqla 0.9.x
*** gokrokve has joined #openstack-infra22:24
flashgordonsdague: have you seen this before:
flashgordonno space left on disk22:27
clarkbflashgordon: which disk? the / disk is really small on those images22:28
clarkbflashgordon: we may need to tell more things to use /opt22:28
flashgordonclarkb: not sure which disk22:28
flashgordonbut yeah22:28
*** gokrokve has quit IRC22:29
clarkbhuh it is trying to do it with cgroups but cgroups are a virtual file system22:30
clarkbthey live under /sys22:30
*** BadCub01 has quit IRC22:30
flashgordonohh I think that is a recent patch22:30
clarkbto do cgroups?22:30
*** jepoy has quit IRC22:32
flashgordonI am mistaken22:32
sdagueflashgordon: nope, that's new to me22:32
sdagueit's nice of libvirt to tell us which device there was no space on ....22:32
sdagueterrible logs are terrible22:33
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Fix syntax error in fedora nodepool config
jeblairclarkb: urgent ^22:33
flashgordonhaha is that your new mantra22:33
flashgordonsdague: like counting is hard22:33
flashgordon(which it is)22:33
*** harlowja_still_a has quit IRC22:33
*** wchrisj has quit IRC22:34
flashgordonmessage:"cgroup\: No space left on device" AND filename:logs*screen-n-cpu.txt22:34
flashgordonlooks like a new bug22:34
clarkbjeblair: looking22:34
flashgordonmy guess is new libvirt patch22:34
flashgordonbased on rough timing22:34
clarkbjeblair: did you catch the comment about the logs fs? it is getting near full again22:34
sdagueflashgordon: did it show up with the new libvirt?22:34
*** enikanorov__ has quit IRC22:34
sdaguebasically did we trade one bug for another22:34
clarkbjeblair: approved22:35
jeblairclarkb: that slope definitely seems to have increased22:35
flashgordonsdague: only has hits in last 24 hours22:35
sdagueand, honestly, since when were we running cgroups22:35
sdagueflashgordon: then probably yes22:36
clarkbjeblair: yeah I think it may be related to the recent logstash size increase too22:36
sdagueflashgordon: so target it as an ubuntu bug as well22:36
* flashgordon files a bug22:36
clarkbjeblair: I swaer I looked not long ago and it was 8.5TB22:36
flashgordondims: ^^^22:36
sdagueand explain we're running serge's hot fix22:36
clarkbabout 2 weeks ago22:36
jeblairclarkb, sdague, flashgordon: did you get to the bottom of/fix the 'we have lots more logs possibly related to ceilometer' issue?22:37
clarkbjeblair: sort of22:37
clarkbit is still doing it though22:37
clarkbbecause ceilometer polls swift22:37
sdagueclarkb: is this the log server, or logstash?22:37
clarkbsdague: log server22:37
clarkbsdague: went from 8.5TB to 11TB in two weeks22:37
sdagueok, so the log server is definitely being pummled by ceilometer22:37
clarkbI think so /me looks at more numbers22:37
flashgordonjeblair: it was a two fold issue. ceilometer middleware in swift was spamming things. and secondly ceilometer is polling swift every 15 seconds22:38
flashgordonwe still have the 15 second poll from ceilo, so swift is generating a lot more logs22:38
clarkbsdague: see
sdaguethe conductor logs are way up again as well22:38
clarkbare we also grabbing /etc/ stuff?22:39
clarkbI expect those to be small but maybe some are large22:39
jeblairclarkb: do you recall the per-host cinder limit?22:40
sdagueetc is super small22:40
jeblairclarkb: this page says 14 volumes
jeblairwe have 1322:40
flashgordonsdague: for conductor logs anything pop out?22:40
clarkbjeblair: yeah we are right up against the limits right now22:40
sdagueflashgordon: all of it22:40
openstackgerritA change was merged to openstack-infra/config: Fix syntax error in fedora nodepool config
sdaguethe conductor logs are basically a trace of all of amqp22:41
sdaguethey really need a diet22:41
sdagueit's 99.99999% debug22:41
*** rpodolyaka1 has joined #openstack-infra22:41
sdagueso the top 6 logs by size22:43
sdague[   ]screen-ceilometer-anotification.txt.gz21-Mar-2014 18:101.2M22:43
sdague[   ]tempest.txt.gz21-Mar-2014 18:101.7M22:43
sdague[   ]screen-n-api.txt.gz21-Mar-2014 18:102.1M22:43
sdague[   ]screen-ceilometer-acentral.txt.gz21-Mar-2014 18:102.3M22:43
sdague[   ]screen-n-cpu.txt.gz21-Mar-2014 18:102.4M22:43
sdague[   ]screen-n-cond.txt.gz21-Mar-2014 18:105.2M22:43
flashgordonsdague: do we need to just turn off amqp debug log evel?22:43
sdague[   ]screen-ceilometer-collector.txt.gz21-Mar-2014 18:106.3M22:43
sdague7 actually22:43
sdagueflashgordon: we already do22:43
uvirtbotLaunchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New]22:43
sdaguethen conductor just logs it itself again22:43
sdaguehonestly, some sifting would be good22:43
sdaguemy tempest patch series that's up cuts the tempest log in half22:43
dimsflashgordon, this was the diff of the patched libvirt -
sdagueclarkb: I need to call it a week, my wife is home, and it's been a long one22:44
jeblairclarkb: i'm adding another (presumably the last) volume to static22:44
clarkbjeblair: roger22:44
jeblairclarkb: we also have about 250g of unallocated space on existing volumes22:44
sdaguehowever if you wanted to start an ML thread on this issue22:44
clarkbsdague: sure why don't I do that while jeblair does the technical stuff22:45
sdagueespecially calling out the worst offenders22:45
flashgordondims: that patch means nothing to me I don't know the context for the code22:45
sdagueand saying "big logs kill puppies, or why you can't have nice things if you don't help us trim these down."22:45
jeblairclarkb, sdague: next step for us is likely reducing retention to < 6 mo; at least until the swift stuff is in place22:45
jeblair(which is close)22:45
jeblair(but is going to take a lot of testing and work so still isn't a near term fix)22:46
sdaguejeblair: well, dealing with the compressed retrevial is still unsolved right?22:46
sdagueI feel like that's not a solved problem yet22:46
flashgordondims: so it looks like this new bug is much less frequent (I think)22:46
jeblairsdague: that's correct22:47
*** jasondotstar has joined #openstack-infra22:47
*** rpodolyaka1 has quit IRC22:47
flashgordondims: in last 12 hours only 10 builds hit this22:47
jeblairsdague: enjoy your weekend!22:48
sdagueyou too22:48
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1295876
uvirtbotLaunchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New]
flashgordonclarkb: can you include a pretty graph in your email?22:49
flashgordondims so the new bug is much less frequebnt22:50
flashgordonclarkb: thanks!22:51
*** alexpilotti has quit IRC22:52
dimsflashgordon, be with you in 10 mins or so22:52
*** mallow123 has quit IRC22:54
jeblairclarkb: resizing logs volume (adding 1024g)22:54
*** nati_ueno has joined #openstack-infra22:54
clarkboh also, the subunit logs for some projects are huge I bet22:55
clarkbflashgordon: can you spot check nova, neutron, cinder, etc subunit log files for me?22:55
flashgordonclarkb: sure what do you want me to check for?22:57
*** jasondotstar has quit IRC22:57
clarkbflashgordon: file size22:57
flashgordonbtw just cut hacking 0.8.122:57
clarkba list like sdague's aboev for the different projects would be great22:57
flashgordonhopefully that didn't break anything ...22:58
flashgordonfrom top of gate22:58
*** dims has quit IRC22:59
flashgordonnothing is that big in there22:59
clarkbflashgordon: not for tempest22:59
clarkbfor unittests22:59
flashgordonchecking full22:59
*** zhiwei has joined #openstack-infra22:59
clarkbso python26 and python27 jobs23:00
*** e0ne has joined #openstack-infra23:00
flashgordonnova: 14MB23:01
flashgordonthat was my fault too:)23:01
flashgordonI enabled debug logs23:01
flashgordonsorry :/23:01
jeblairwow, watching the used space on static.o.o is like watching the national debt clock23:01
flashgordon14 MB for gate-nova-python2623:02
clarkbflashgordon: lolololol23:02
clarkbjeblair: :(23:02
*** CaptTofu has joined #openstack-infra23:02
flashgordoncinder: <1MB23:02
clarkbflashgordon: I think neutron will be massive23:02
*** gokrokve has joined #openstack-infra23:03
flashgordon7.8 MB23:03
flashgordonfor  both 26 2723:04
flashgordonglance 86K23:04
flashgordonkeysone 4.8MB
clarkbflashgordon: yeah so I think neutron and nova are the offenders here23:04
*** e0ne has quit IRC23:05
flashgordonceilo 158K23:05
flashgordonswift 1.423:05
flashgordonclarkb: agreed23:05
flashgordonwith keystone in #323:05
flashgordon Ijust checked integrated projects23:05
clarkbhow does that etherpad look?23:08
flashgordonlooks good23:09
flashgordonmaybe mention icehouse release23:10
lifelessclarkb: jeblair indicated it wasn't affecting nodepool other than not having as many slaves as desired so I ignored in favor of family23:10
*** esker has joined #openstack-infra23:10
*** BadCub01 has joined #openstack-infra23:10
lifelessclarkb: also we need to capture data for nova to fix it23:10
clarkblifeless: ++ to ignoring for family. It does also slightly affect our ability ot build images though whcih isn't a huge deal wither23:10
flashgordonpretty impressive23:10
*** jasondotstar has joined #openstack-infra23:11
clarkbjeblair: flashgordon I am calling that etherpad good and will send it shortly if you guys don't have anything to add23:11
flashgordonclarkb: maybe add a source for file sizes23:12
flashgordonI was using;O=D23:12
flashgordonwhich has similar results23:12
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add the ability to create subnodes
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive
jeblair/dev/mapper/main-logs          12T   11T  1.7T  86% /srv/static/logs23:14
jeblairclarkb: complete ^23:14
jeblairclarkb: we still have 250g unallocated23:14
*** dims has joined #openstack-infra23:14
*** denis_makogon_ has joined #openstack-infra23:15
*** vkozhukalov has quit IRC23:15
flashgordonclarkb: for nova I am thinking turn off oslo.messaging debug logs?23:16
dimsflashgordon, 1295876 and the large logs are unrelated, right?23:16
flashgordondims: yes23:16
clarkbjeblair: thanks23:16
clarkbjeblair: does the email look good to you?23:17
jeblairclarkb: just confirmed that we can't add more volumes to static23:17
jeblairclarkb: lgtm23:18
* clarkb does last minute formatting and will send23:19
*** gokrokve has quit IRC23:21
flashgordonclarkb: gonna see what does to nova logs23:21
*** gokrokve has joined #openstack-infra23:21
*** mriedem has quit IRC23:22
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support
jeblairokay, that's the whole patch series for nodepool multi-node23:23
clarkbflashgordon: jeblair sdague sent the message23:24
clarkbjeblair: that includes the third step now?23:24
clarkbjeblair: I will try to review the stack again after beer thirty which I am being called away to23:24
jeblairclarkb: yep23:24
jeblairyeah, i'm going to go run around in circles for a while23:24
jeblairwe have 31/33 new images23:25
*** gokrokve has quit IRC23:25
clarkbjeblair: looking back at the cacti graph, I wonder if maybe some things aren't getting deleted? flashgordon's example from september above doesn't support that theory though23:26
clarkbsince it was properly deleted23:26
clarkbbut time for me to afk for a bit23:26
*** mrodden has quit IRC23:26
*** caleb_ has joined #openstack-infra23:29
*** hogepodge has left #openstack-infra23:33
*** CaptTofu has quit IRC23:38
*** wchrisj has joined #openstack-infra23:38
*** asselin has joined #openstack-infra23:38
*** jepoy has joined #openstack-infra23:41
*** jasondotstar has quit IRC23:43
fungiclarkb: jeblair: caught up now... yes, 14 cinder volumes because the version of xen rackspace is running can't present more than 16 virtual block devices to a domu (and root and swap are already occupying two of those)23:43
*** david_lyle_ has quit IRC23:44
*** rpodolyaka1 has joined #openstack-infra23:44
fungiand their cinder volume size maximum is 1tb, so 14tb of cinder space max per instance23:44
fungialso, the retention reduction would have to be substantially less than 6 months because out growth rate means that something like 75% of the logs will be from just the past few months23:45
fungiwe could run some estimates, but i expect we'd need to cut our retention to something like 2-3 months23:45
*** khyati_ has quit IRC23:45
*** CaptTofu has joined #openstack-infra23:47
fungibecause remember that what you're looking at with ethe filesystem graph is a derivative (sliding window of 6 months data total size)23:47
fungi6 months ago, 6 months of data was ~3tib. so the rate at which we accumulate data has more than tripled in size in 6 months23:48
*** rpodolyaka1 has quit IRC23:48
*** dstanek has quit IRC23:48
*** thuc_ has joined #openstack-infra23:50
*** denis_makogon_ has quit IRC23:51
*** Ryan_Lane has joined #openstack-infra23:52
dimsflashgordon, nothing pops out, and it does not seem to be causing a lot of harm to the gate. let's wait a bit and see if it gets worse. ok?23:53
*** thuc has quit IRC23:53
dimsi was worried that the libvirt change caused a flood of logs.23:54
*** thuc_ has quit IRC23:54
*** yamahata has quit IRC23:55
*** yamahata has joined #openstack-infra23:55
*** zzelle has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!