Monday, 2016-02-08

*** FallenPegasus has quit IRC00:10
*** FallenPegasus has joined #openstack-infra00:11
*** sdake has quit IRC00:12
*** darrenc_afk is now known as darrenc00:13
*** hichihara has joined #openstack-infra00:13
*** amotoki has joined #openstack-infra00:16
*** davideagnello has joined #openstack-infra00:18
*** salv-orl_ has quit IRC00:19
*** gildub has joined #openstack-infra00:19
*** jsavak has joined #openstack-infra00:23
*** davideagnello has quit IRC00:23
*** achanda has joined #openstack-infra00:25
*** chlong has joined #openstack-infra00:26
*** hamzy has joined #openstack-infra00:26
*** peterlisak_ has joined #openstack-infra00:29
*** amaretskiy1 has joined #openstack-infra00:29
*** clif has joined #openstack-infra00:30
*** FallenPe_ has joined #openstack-infra00:33
*** amaretskiy has quit IRC00:33
*** peterlisak has quit IRC00:33
*** clif_h has quit IRC00:33
*** FallenPegasus has quit IRC00:33
*** peterlisak_ is now known as peterlisak00:33
*** gildub has quit IRC00:33
*** tiswanso has quit IRC00:37
*** FallenPe_ has quit IRC00:37
*** mikal is now known as chair_occupier_t00:41
*** chair_occupier_t is now known as chair_occupier_300:41
*** FallenPegasus has joined #openstack-infra00:44
*** davideagnello has joined #openstack-infra00:45
SamYapleclarkb: re docker image per commit: It would need to be every image every commit (rather than a subset). If we don't do it per commit than what would happen is master would never have working images in dockerhub, but releases would00:48
SamYaplethat may work too, but just so you are aware.00:48
SamYapleA nightly push is reasonable though. so images may be out of date for a small portion of a day _sometimes_00:48
SamYaplefor stable branches a per tag push is fine00:49
*** jsavak has quit IRC00:50
*** davideagnello has quit IRC00:50
*** jsavak has joined #openstack-infra00:50
*** gildub has joined #openstack-infra00:50
*** fedexo has quit IRC00:50
*** gildub has quit IRC00:58
*** dims has quit IRC01:01
*** chlong has quit IRC01:03
*** FallenPegasus has quit IRC01:03
*** amrith is now known as _amrith_01:09
*** davideagnello has joined #openstack-infra01:12
*** tzn has quit IRC01:14
*** jsavak has quit IRC01:15
*** gildub has joined #openstack-infra01:15
*** jsavak has joined #openstack-infra01:17
*** davideagnello has quit IRC01:17
*** salv-orlando has joined #openstack-infra01:19
*** dims has joined #openstack-infra01:24
openstackgerritEmilien Macchi proposed openstack-infra/project-config: run integration jobs for puppet-gnocchi
openstackgerritEmilien Macchi proposed openstack-infra/project-config: add tripleo job to test puppet-ceph
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a devstack test job using Congress new architecture
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture
*** tzn has joined #openstack-infra01:39
*** davideagnello has joined #openstack-infra01:40
*** davideagnello has quit IRC01:44
*** tzn has quit IRC01:44
*** dims has quit IRC01:46
*** yamamoto_ has joined #openstack-infra01:48
*** katyafervent_awa has quit IRC01:50
*** salv-orlando has quit IRC01:50
*** katyafervent_awa has joined #openstack-infra01:53
*** jsavak has quit IRC02:02
*** chlong has joined #openstack-infra02:06
*** davideagnello has joined #openstack-infra02:07
*** yamamoto_ has quit IRC02:07
*** yamamoto_ has joined #openstack-infra02:08
*** yamamoto_ has quit IRC02:10
*** yamamoto_ has joined #openstack-infra02:10
*** yamamoto_ has quit IRC02:10
*** davideagnello has quit IRC02:11
*** daemontool has quit IRC02:12
*** daemontool has joined #openstack-infra02:13
*** tiswanso has joined #openstack-infra02:16
*** tiswanso has quit IRC02:21
funginibalizer: yeah, i'm planning to try and set up an account with startssl in the next couple weeks, and i'll get a new cert for it there. namecheap worked well enough when we only had a handful of certs, but it's getting cumbersome now and it looks like startssl will let us get an unlimited number of certs generated whenever we want for a flat rate (less than i'm paying for all our current certs too)02:32
*** davideagnello has joined #openstack-infra02:34
clarkbfungi death wish?02:34
clarkbtheir website stopped responding, I assume the ad worked02:35
fungiyes, and not surprised :/02:35
mordredclarkb: they had a sportsball ad?02:36
openstackgerritMerged openstack/requirements: Bump minimum tempest-lib version
fungithey won a contest intuit was running to sponsor a small business's national superbowl ad02:37
mordredclarkb: (I did not watch the first half - I was shopping for furniture instead - turns out during superbowl is a GREAT time to go to a very large furniture store :) )02:37
mordredfungi: I thought that was your coffee ...02:37
mordredI mixed the convo about startssl and the convo about the coffee02:37
mordredI thought you were saying the startssl site was down due to a superbowl ad02:38
mordredand I was like "dude - startssl did a superbowl ad?"02:38
* mordred is caught up now02:38
fungilack of threading in irc ;)02:38
clarkbmordred: I recommend watching sportball during sportball02:38
mordredclarkb: indeed02:38
clarkbits been a decent game too02:38
*** davideagnello has quit IRC02:38
mordredclarkb: yah - second half is nice02:38
mordredclarkb: however, we just moved into this place that is not the size of a new york apartment, so basically feel like we have no furniture at all02:39
clarkbjust amazon 2day some stuff02:39
clarkbwe did this for all baby furniture02:39
mordredI think amazoning a sectional sofa might be a bit much02:40
mordred(want to sit/lay on things)02:40
pleia2mordred: congrats on the new place (and furniture!)02:40
mordredpleia2: thanks!02:40
*** FallenPegasus has joined #openstack-infra02:40
* mordred is torn between wanting to invite everyone over for an afterstack ... and imagining having EVERYONE over for an afterstack02:40
clarkbmordred: people do it, beds too02:40
*** tzn has joined #openstack-infra02:41
fungiluckily when we bought, the place came completely furnished with pretty nice stuff (yay vacation town), so the fact that we were moving from a tiny place and were bringing almost no furniture actually worked out very well02:41
* greghaynes got his bed off Amazon02:41
mordredfungi: nice!02:41
mordredfungi: we did get a table with the place02:41
greghaynesBut I have an Amazon problem02:41
mordredthe previous owners had a custom table made from the same granite as the countertops02:41
clarkbI should buy a place02:41
fungigreghaynes: admitting it is the first step02:41
mordredso taking it with them didn't really make any sense02:41
clarkbit turns out thisnis difficult02:41
clarkbportland world problems02:42
greghaynesfungi: its a long process02:42
mordredclarkb: there is a house for sale 2 down from me02:42
pleia2clarkb: heh, yeah, I was going to say you should do it soon since house prices rising up there02:42
*** changbl has quit IRC02:42
mordredclarkb: probably half the price of all of the houses in portland, fwiw02:42
clarkbI actually had an offer accepted02:42
clarkbthen sellers got crazy and weird02:42
greghaynesWe both did...02:42
clarkband we noped ouy02:42
greghaynesPortland housing is not fun02:42
*** changbl has joined #openstack-infra02:42
mordredsellers suck02:42
*** oomichi has quit IRC02:43
clarkbit was really bad02:43
clarkbthey started demanding more money and not communicatimg02:43
mordredoh neat02:43
mordredthat's exactly what you want02:43
clarkbso I called and they can find anew buyer02:43
clarkb(we apparently pissed them off when we asked them to correct the siding which was also their first item on all advertising with "new siding!!!?")02:44
clarkbbut was installed completely wrong02:45
clarkband it all went downhill from there02:45
pleia2ugh, that's awful02:45
clarkbat this rate I might find a cardboard shack under a bridge02:46
clarkbat least that will come with river views02:46
*** daemontool has quit IRC02:47
pleia2we bought in 2009, which is probably the smartest thing we've ever done02:47
*** akshai has joined #openstack-infra02:48
*** [1]Thelo has joined #openstack-infra02:51
*** Thelo has quit IRC02:53
*** [1]Thelo is now known as Thelo02:53
*** akshai has quit IRC02:54
*** yamamoto_ has joined #openstack-infra02:58
*** davideagnello has joined #openstack-infra03:01
*** davideagnello has quit IRC03:06
*** salv-orlando has joined #openstack-infra03:10
mordredclarkb: there was that time we were in portland and walked by that nice mattress on the bridge underpass - I bet you could take whoever owns that03:11
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels
*** salv-orlando has quit IRC03:13
*** dims has joined #openstack-infra03:13
*** tzn has quit IRC03:14
*** rguillebert has quit IRC03:17
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Add an experiment job for Neutron FWaaS tempest tests
*** unicell has quit IRC03:22
*** unicell has joined #openstack-infra03:22
openstackgerritAngus Lees proposed openstack-infra/project-config: Create new tempest-dsvm-full-head job-template
*** gildub has quit IRC03:25
*** mingyu has joined #openstack-infra03:27
*** mingyu has quit IRC03:28
*** chair_occupier_3 is now known as mikal03:34
*** dims has quit IRC03:35
*** tiswanso has joined #openstack-infra03:36
*** daemontool has joined #openstack-infra03:38
*** dims has joined #openstack-infra03:42
*** davideagnello has joined #openstack-infra03:42
*** dkranz has quit IRC03:44
*** davideagnello has quit IRC03:46
SamYapleoh.... i guess the superbowl was tonight03:48
openstackgerritTim Burke proposed openstack-dev/hacking: Run all flake8 checks
*** gildub has joined #openstack-infra03:58
*** baoli has joined #openstack-infra04:05
*** hichihara has quit IRC04:06
*** dingyichen has quit IRC04:08
*** dims has quit IRC04:08
openstackgerritTim Burke proposed openstack-dev/hacking: Add H203 to check that assertIs(Not)None is used
*** tzn has joined #openstack-infra04:11
*** dingyichen has joined #openstack-infra04:13
*** amotoki has quit IRC04:13
*** hichihara has joined #openstack-infra04:13
*** links has joined #openstack-infra04:14
*** tzn has quit IRC04:16
tonyblifless: What's the correct workflow for sugesting an enhancement in fixtures?  is it all github based?04:18
tonyblifeless: note I'm not 100% certain it is a fixtures issue but if it is I'm looking at how to fix it ...04:19
*** mingyu has joined #openstack-infra04:21
*** davideagnello has joined #openstack-infra04:23
*** amotoki has joined #openstack-infra04:27
*** davideagnello has quit IRC04:27
*** david-lyle has quit IRC04:30
*** dstanek has quit IRC04:31
*** dstanek has joined #openstack-infra04:33
*** tiswanso has quit IRC04:33
*** hdd has joined #openstack-infra04:34
*** unicell has quit IRC04:37
*** unicell has joined #openstack-infra04:37
*** eliqiao_ has quit IRC04:38
*** andrearosa has quit IRC04:38
*** andrearosa has joined #openstack-infra04:38
*** lennyb_ has quit IRC04:38
*** lennyb_ has joined #openstack-infra04:38
*** amotoki has quit IRC04:39
*** eliqiao_ has joined #openstack-infra04:40
*** amotoki has joined #openstack-infra04:40
*** salv-orlando has joined #openstack-infra04:44
*** achanda has quit IRC04:45
*** david-lyle has joined #openstack-infra04:45
*** baoli has quit IRC04:48
*** baoli has joined #openstack-infra04:50
*** dimtruck is now known as zz_dimtruck04:50
*** davideagnello has joined #openstack-infra04:50
*** kzaitsev_mb has joined #openstack-infra04:50
*** baoli has quit IRC04:51
*** davideagnello has quit IRC04:54
*** chaitu has quit IRC04:55
openstackgerritIan Wienand proposed openstack/diskimage-builder: Revert "Skip centos functional testing"
*** amotoki has quit IRC04:56
*** salv-orlando has quit IRC04:56
*** shashank_hegde has quit IRC04:57
*** hdd has quit IRC05:03
*** eliqiao_ has quit IRC05:06
*** eliqiao_ has joined #openstack-infra05:08
*** amotoki has joined #openstack-infra05:11
*** tzn has joined #openstack-infra05:12
*** hdd has joined #openstack-infra05:17
*** Sukhdev has joined #openstack-infra05:17
*** davideagnello has joined #openstack-infra05:17
*** tzn has quit IRC05:17
*** nikil89_ has joined #openstack-infra05:19
*** achanda has joined #openstack-infra05:21
*** davideagnello has quit IRC05:22
*** hdd has quit IRC05:22
*** hdd has joined #openstack-infra05:34
*** hdd has quit IRC05:37
lifelesstonyb: issues in launchpad, ocde in github05:39
tonyblifeless: okay thanks.05:39
* tonyb just wasted have a day on a related issue only to find the test "works" on MacOS but fails on Linux *sigh*05:40
*** abregman has joined #openstack-infra05:42
lifelesstonyb: :/05:43
lifelesstonyb: what was the issue?05:43
tonyblifeless: I'm trying to reproduce a failure seen in the gate post testtoosl 2.0.0 release.05:44
*** davideagnello has joined #openstack-infra05:44
tonyblifeless: the fact the mac worked has somethign to do with it trying to use berkleyDB support.05:45
tonyblifeless: AFAICT that failing altered the code that was being run so I never got to the actual code I wanted to test :(05:45
tonyblifeless: copying the same source and reproducing the venv in linux and I trigger the bug right away.05:46
tonyblifeless: so now to the testtools/fixtures issue05:46
lifelesstonyb: is there a bug report?05:46
tonyblifeless: on the testtools/fixtures issue?05:46
lifelesstonyb: either :)05:46
tonyblifeless: I'm not going to bother the the db thing, and I'll do a bug report for the otehr once I confirm it :)05:47
*** davideagnello has quit IRC05:49
openstackgerritIan Wienand proposed openstack/diskimage-builder: Switch simple-init to pip-and-virtualenv element
openstackLaunchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New]06:01
lifelesstonyb: thats going to be closed not a bug06:01
lifelesstonyb: 2.0.0 depends on 1.3.0 or greater06:01
tonyblifeless: It's probably a little rambly as I created it wheil watching the kids in the bath.06:01
lifelesstonyb: and our metadata expresses that06:01
tonybright but fixtures 1.2.0 allows 2.0.0 to be installed06:01
lifelesstonyb: not as such06:02
lifelesstonyb: you're running into a known bug in pip I suspect; I'll look once Cynthia is in bed06:02
tonyblifeless: okay.06:02
tonyblifeless: I inclued links to the repro in tha gate06:02
tonyblifeless: I'd be happy for any solution.06:03
tonyblifeless: I'll be back once my kids are in bed06:03
*** amotoki_ has joined #openstack-infra06:08
*** amotoki has quit IRC06:11
*** davideagnello has joined #openstack-infra06:11
*** tzn has joined #openstack-infra06:14
*** davideagnello has quit IRC06:16
lifelesstonyb: k; I have replied in brief there06:16
lifelesstonyb: happy to discuss more, and perhaps do a pragmatic if ick thing, but we'll see06:17
openstackgerritIan Wienand proposed openstack-infra/puppet-nodepool: Fix cleanup of rotated files
*** tzn has quit IRC06:19
*** naresht has joined #openstack-infra06:20
nareshtHi all. I developed an application for openstack/murano-apps. I want to submit for code review. openstack/murano-apps uses Gerrit for code review. How can I do this?06:23
*** chaitu has joined #openstack-infra06:28
*** baoli has joined #openstack-infra06:28
chaituwhile starting nodepool services for the first time by default it is starting using option as --no-builder. how could i start nodepool service with builder
*** baoli has quit IRC06:33
lifelesstonyb: wb06:37
*** davideagnello has joined #openstack-infra06:39
*** davideagnello has quit IRC06:43
*** Daisy has joined #openstack-infra06:45
*** unicell has quit IRC06:45
*** salv-orlando has joined #openstack-infra06:48
*** unicell has joined #openstack-infra06:49
*** Daisy has quit IRC06:49
*** savihou has joined #openstack-infra06:50
*** Daisy has joined #openstack-infra06:50
*** daemontool has quit IRC06:53
*** daemontool has joined #openstack-infra06:54
*** Daisy has quit IRC06:55
*** salv-orlando has quit IRC06:55
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** gildub has quit IRC06:56
*** daemontool has quit IRC06:58
*** chlong has quit IRC07:03
*** davideagnello has joined #openstack-infra07:06
*** kzaitsev_mb has quit IRC07:08
*** davideagnello has quit IRC07:10
*** oomichi has joined #openstack-infra07:12
openstackgerritFredrik Medley proposed openstack-infra/zuul: Improve job dependencies using graph instead of tree
*** tzn has joined #openstack-infra07:14
*** jtomasek__ has joined #openstack-infra07:15
*** tzn has quit IRC07:19
*** Jeffrey4l has joined #openstack-infra07:19
*** infra-red has joined #openstack-infra07:20
*** jtomasek_ has joined #openstack-infra07:25
*** Daisy has joined #openstack-infra07:26
openstackgerritMerged openstack/diskimage-builder: Switch simple-init to pip-and-virtualenv element
*** davideagnello has joined #openstack-infra07:33
*** kushal has joined #openstack-infra07:34
*** davideagnello has quit IRC07:38
openstackgerritMerged openstack-infra/project-config: Change shade keystone v2 job for stable/liberty
*** Daisy has quit IRC07:42
*** Daisy has joined #openstack-infra07:43
*** Daisy has quit IRC07:46
*** hichihara has quit IRC07:48
*** zeih_ has joined #openstack-infra07:48
*** flepied has quit IRC07:49
*** nmagnezi has joined #openstack-infra07:49
*** jaosorior has joined #openstack-infra07:57
SamYaplemorning yolanda07:59
*** davideagnello has joined #openstack-infra08:00
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud
*** jlanoux has joined #openstack-infra08:02
*** davideagnello has quit IRC08:05
*** jaosorior has quit IRC08:05
*** |-paul-| has joined #openstack-infra08:05
*** salv-orlando has joined #openstack-infra08:07
*** rcernin has joined #openstack-infra08:09
*** scheuran has joined #openstack-infra08:09
*** salv-orlando has quit IRC08:12
openstackgerritMerged openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee
*** samuelBartel has joined #openstack-infra08:13
*** Sukhdev has quit IRC08:13
*** jaosorior has joined #openstack-infra08:13
openstackgerritSpencer Krum proposed openstack-infra/system-config: Script to generate dns for infracloud
*** arxcruz has joined #openstack-infra08:16
*** tzn has joined #openstack-infra08:16
*** flepied has joined #openstack-infra08:16
openstackgerritMerged openstack-infra/project-config: run integration jobs for puppet-gnocchi
openstackgerritMerged openstack-infra/project-config: add tripleo job to test puppet-ceph
openstackgerritMerged openstack-infra/project-config: Adds a devstack test job using Congress new architecture
openstackgerritMerged openstack-infra/project-config: Create new tempest-dsvm-full-head job-template
*** rubasov has joined #openstack-infra08:19
*** sdake has joined #openstack-infra08:20
*** tzn has quit IRC08:20
openstackgerritMerged openstack-infra/project-config: Adds a tox test job for Congress new architecture
openstackgerritFrank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite
*** ifarkas has joined #openstack-infra08:24
*** davideagnello has joined #openstack-infra08:27
*** Daisy has joined #openstack-infra08:29
*** achanda has quit IRC08:31
openstackgerritAlistair Coles proposed openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post
*** davideagnello has quit IRC08:32
*** jistr has joined #openstack-infra08:33
*** jistr is now known as jistr|sick08:33
openstackgerritMerged openstack-infra/project-config: Don't run gate-sahara-tox-bandit on stable/kilo
openstackgerritMerged openstack-infra/project-config: Remove Nova pip-check-reqs job
openstackgerritMerged openstack-infra/project-config: Add grenade testing for oslo.log
openstackgerritMerged openstack-infra/infra-manual: Creator's Guide: Add some pointer
*** yaume has joined #openstack-infra08:40
*** fhubik has joined #openstack-infra08:42
openstackgerritVasyl Saienko proposed openstack/requirements: Add netmiko to requirements
openstackgerritMerged openstack-infra/project-config: Import ansible-roles-jenkins(-plugins) projects
*** fhubik is now known as fhubik_brb08:43
openstackgerritMerged openstack-infra/puppet-project_config: Add nodepool.yaml to the layout
*** achanda has joined #openstack-infra08:44
*** mrmartin has joined #openstack-infra08:44
*** dingyichen has quit IRC08:46
openstackgerritMerged openstack-infra/puppet-zuul: Allow configuration of X-Account-Meta-Temp-Url-Key
*** chlong has joined #openstack-infra08:48
*** Hal has joined #openstack-infra08:49
*** Hal is now known as Guest4807408:50
*** fhubik_brb is now known as fhubik08:50
*** davideagnello has joined #openstack-infra08:55
*** kzaitsev_mb has joined #openstack-infra08:55
*** amotoki_ has quit IRC08:55
*** jaosorior has quit IRC08:57
*** davideagnello has quit IRC08:59
*** openstackgerrit has quit IRC09:02
*** openstackgerrit has joined #openstack-infra09:02
*** jaosorior has joined #openstack-infra09:03
openstackgerritVasyl Saienko proposed openstack/requirements: Add netmiko to requirements
*** fhubik is now known as fhubik_brb09:06
*** lezbar has joined #openstack-infra09:07
*** adrienverge has joined #openstack-infra09:07
*** achanda has quit IRC09:08
*** lucas-dinner is now known as lucasagomes09:09
*** fhubik_brb is now known as fhubik09:12
*** zeih_ has quit IRC09:13
*** amotoki has joined #openstack-infra09:14
*** matrohon has joined #openstack-infra09:17
*** tzn has joined #openstack-infra09:17
*** rossella_s has joined #openstack-infra09:19
*** tzn has quit IRC09:21
*** davideagnello has joined #openstack-infra09:21
*** yuanying is now known as yuanying_influen09:25
*** yuanying_influen is now known as yuanying_flu09:25
*** davideagnello has quit IRC09:26
*** hashar has joined #openstack-infra09:27
*** zeih_ has joined #openstack-infra09:27
*** isaacb has joined #openstack-infra09:28
*** fhubik is now known as fhubik_brb09:29
*** salv-orlando has joined #openstack-infra09:31
*** aarefiev_ has quit IRC09:31
*** amotoki has quit IRC09:32
*** aarefiev has joined #openstack-infra09:32
*** ildikov has quit IRC09:37
*** Jeffrey4l has quit IRC09:37
*** dguitarbite has quit IRC09:45
*** mrmartin has quit IRC09:48
*** fhubik_brb is now known as fhubik09:49
*** tzn has joined #openstack-infra09:50
*** kzaitsev_mb has quit IRC09:50
*** tzn has quit IRC09:51
*** ihrachys has joined #openstack-infra09:51
*** tzn has joined #openstack-infra09:52
*** haypo has joined #openstack-infra09:55
haypoianw: hello. i tried to use env markers in requirements of the gnocchi project, but tox -e bashate failed because bashate 0.3.2 (latest release) still requires pbr <1.009:56
openstackgerritJulien Danjou proposed openstack/requirements: Add overtest to global-req
haypoianw: would you be ok to release a new version of bashate?09:56
haypojd__, ianw : ah, it looks like bashate doesn't use ?09:57
jd__haypo: which team does it belong to?09:58
*** dtantsur|afk is now known as dtantsur09:58
*** pedroalvarez_ is now known as pedroalvarez09:58
jd__QA it seems09:58
jd__and it's marked release:independent09:59
jd__haypo: so you have to ask mtreinish probably to do a release, or someone in that team09:59
haypomtreinish: hey! i need a release for bashate :) it's blocking me to "fix" requirements of gnocchi,
*** rossella_s has quit IRC10:03
*** rossella_s has joined #openstack-infra10:03
*** flepied has quit IRC10:04
openstackgerritFrank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite
*** kzaitsev_mb has joined #openstack-infra10:06
*** placko has joined #openstack-infra10:07
*** ildikov has joined #openstack-infra10:10
*** rguillebert has joined #openstack-infra10:14
*** sfinucan has joined #openstack-infra10:15
*** _nadya_ has joined #openstack-infra10:15
*** pavel_bondar has quit IRC10:17
*** electrofelix has joined #openstack-infra10:18
*** Daisy has quit IRC10:20
*** flepied has joined #openstack-infra10:21
*** flepied1 has joined #openstack-infra10:24
*** isaacb has quit IRC10:24
*** flepied has quit IRC10:27
*** ildikov has quit IRC10:28
*** gildub has joined #openstack-infra10:29
*** Daisy has joined #openstack-infra10:29
*** isq_ has quit IRC10:30
*** ociuhandu has quit IRC10:30
*** isq_ has joined #openstack-infra10:31
*** ildikov has joined #openstack-infra10:36
*** mingyu has quit IRC10:39
*** salv-orl_ has joined #openstack-infra10:40
*** kzaitsev_mb has quit IRC10:41
*** salv-orlando has quit IRC10:43
*** tpsilva has joined #openstack-infra10:49
*** e0ne has joined #openstack-infra10:53
*** aysyd has joined #openstack-infra10:56
*** davideagnello has joined #openstack-infra10:57
AJaegerhaypo:,members is the list of cores to ask10:57
*** chaitu has quit IRC10:59
*** aysyd has quit IRC10:59
*** fhubik is now known as fhubik_brb11:00
*** aysyd has joined #openstack-infra11:00
*** sdake has quit IRC11:00
*** davideagnello has quit IRC11:02
haypoAJaeger: thanks. email sent11:08
*** electrofelix has quit IRC11:11
*** electrofelix has joined #openstack-infra11:12
*** FallenPegasus has quit IRC11:15
*** mingyu has joined #openstack-infra11:17
*** FallenPegasus has joined #openstack-infra11:18
*** esikachev has joined #openstack-infra11:19
*** rcernin has quit IRC11:19
*** derekh has joined #openstack-infra11:20
*** rcernin has joined #openstack-infra11:22
*** rcernin has quit IRC11:25
*** rcernin has joined #openstack-infra11:26
*** yamamoto_ has quit IRC11:27
*** Jeffrey4l has joined #openstack-infra11:29
*** zeih_ has quit IRC11:36
*** esikachev has quit IRC11:36
*** jistr|sick has quit IRC11:37
*** davideagnello has joined #openstack-infra11:37
*** dguitarbite has joined #openstack-infra11:38
*** kzaitsev_mb has joined #openstack-infra11:38
*** dguitarbite has quit IRC11:40
*** rhallisey has joined #openstack-infra11:40
*** fhubik_brb is now known as fhubik11:40
*** jaosorior has quit IRC11:41
*** esikachev has joined #openstack-infra11:41
*** zeih_ has joined #openstack-infra11:42
*** davideagnello has quit IRC11:42
*** jaosorior has joined #openstack-infra11:43
*** kzaitsev_mb has quit IRC11:43
*** baoli has joined #openstack-infra11:48
*** Jeffrey4l has quit IRC11:48
*** baoli has quit IRC11:49
*** |-paul-| has quit IRC11:49
*** vgridnev has joined #openstack-infra11:55
*** jaypipes has joined #openstack-infra11:56
*** dguitarbite has joined #openstack-infra11:56
*** dguitarbite has quit IRC11:56
*** ganso has joined #openstack-infra11:58
*** kzaitsev_mb has joined #openstack-infra12:00
AJaegersdague: could you approve nova translation import, please?
*** erlon has joined #openstack-infra12:01
*** dguitarbite has joined #openstack-infra12:02
*** davideagnello has joined #openstack-infra12:05
*** vgridnev has quit IRC12:07
*** lucasagomes is now known as lucas-hungry12:07
*** dims has joined #openstack-infra12:07
*** davideagnello has quit IRC12:10
*** weshay_xchat has quit IRC12:10
*** amotoki has joined #openstack-infra12:13
*** vgridnev has joined #openstack-infra12:13
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing
*** jaosorior has quit IRC12:17
*** salv-orl_ has quit IRC12:18
*** isaacb has joined #openstack-infra12:18
*** tcammann has joined #openstack-infra12:19
*** claudiub has joined #openstack-infra12:19
*** dims has quit IRC12:20
*** gildub has quit IRC12:21
*** yamamoto has joined #openstack-infra12:21
*** dims has joined #openstack-infra12:22
* dims is back. sorry flaky network12:22
*** ociuhandu has joined #openstack-infra12:25
*** gordc has joined #openstack-infra12:30
*** markus_z has joined #openstack-infra12:31
*** davideagnello has joined #openstack-infra12:32
*** amotoki has quit IRC12:35
*** davideagnello has quit IRC12:37
krotscheck_dcmI'm trying to figure out what the job runtimes are for our devstack builds, now that we have the new wheel mirrors in place. Graphite doesn't seem to have ther untimes in there though12:37
krotscheck_dcmANyone familiar with how we've set up graphite, and can tell me if that data's there?12:38
sdagueAJaeger: done12:38
sdaguekrotscheck_dcm: the data is not in graphite12:38
*** miqui has quit IRC12:39
krotscheck_dcmsdague: Hrm.12:39
*** krotscheck_dcm is now known as krotscheck12:39
krotscheckWell, to the logs I go then :)12:39
*** thorst has joined #openstack-infra12:40
sdaguegrabbing a random devstack log, it does look like it's working. We were running about 600s before on pip_install, now it's 18012:41
sdaguethough, looks like a few things don't yet have wheels -
*** salv-orlando has joined #openstack-infra12:42
*** yamamoto has quit IRC12:42
*** yamamoto has joined #openstack-infra12:43
*** weshay_xchat has joined #openstack-infra12:43
*** jed56 has joined #openstack-infra12:44
*** gsagie has joined #openstack-infra12:44
*** jaosorior has joined #openstack-infra12:46
krotschecksdague: Yeah, we encountered an interesting situation on friday where a wheel was built for cpy27, and thus wasn't downloaded.12:47
*** yamamoto has quit IRC12:48
krotschecksdague: MarkupSafe is one of them.
krotscheckI suspect that might be by design? I'ma ssuming that the "c" stands for "I compiled C things", which would be architecture dependent.12:50
*** tcammann has quit IRC12:50
*** [1]Thelo has joined #openstack-infra12:51
*** jaosorior has quit IRC12:51
krotscheckNow that doesn't matter to us, but it could matter to everyone-else-on-the-internet.12:51
krotscheckdstufft  - what does the cpy27 in this wheel indicate?
*** sdake has joined #openstack-infra12:52
dstufftkrotscheck: that wheel is specific to CPython 2.712:52
*** Thelo has quit IRC12:53
*** [1]Thelo is now known as Thelo12:53
krotscheckdstufft: I don't actually understand how that relates to, say, regular python. It was built automatically by our wheel mirror from a list of requirements.12:55
krotscheckdstufft: Should I be looking in setup.cfg? Our build script? Is it something we actually can't control at all?12:56
dstufftkrotscheck: CPython is what most people mean by "Python"12:56
dstufftit's the standard implementation12:56
dstufftalternative implementations are PyPy, Jython, IronPython, etc.12:56
krotscheckGot it. So, I'm guessing, that when pip tries to resolve the wheel, either it's noticing that it's not runing on cpython, or it's not matching the name properly.12:57
krotscheckGiven that we have a repeatable environment here, I suspect it's the latter?12:58
*** amotoki has joined #openstack-infra12:58
*** yamamoto has joined #openstack-infra12:58
dstufftis this running on CPython 2.7?12:58
*** sdake has quit IRC12:58
krotscheckdstufft: The wheel build or the jenkins node?12:59
* krotscheck goes to build a jenknins slave to check12:59
*** abregman has quit IRC12:59
*** FallenPegasus has quit IRC12:59
dstufftkrotscheck: if you can reproduce, -vvv should give more information about why it's not using that wheel too13:00
krotscheckdstufft: I won't be able to fully confirm until this slave builds, but the jenkins slaves and the wheel slaves are cut from the same proveribal cloth. The thing that built it is the thing that consumes it.13:00
dstufftshould be CPython 2.7 then13:01
dstufftI'd run it with -vvv to see what that says13:01
krotscheckRoger wilco13:01
*** daemontool has joined #openstack-infra13:01
*** baoli has joined #openstack-infra13:02
*** dkranz has joined #openstack-infra13:05
*** krotscheck has quit IRC13:06
*** dtardivel has joined #openstack-infra13:06
*** dprince has joined #openstack-infra13:06
*** gal_ has joined #openstack-infra13:06
*** dizquierdo has joined #openstack-infra13:06
*** EricGonczer_ has joined #openstack-infra13:07
*** gsagie has quit IRC13:10
*** _nadya_ has quit IRC13:10
*** EricGonczer_ has quit IRC13:10
*** krotscheck1 has joined #openstack-infra13:11
* krotscheck1 peers at krotscheck.13:12
*** lucas-hungry is now known as lucasagomes13:12
*** gal_ is now known as gsagie13:12
*** davideagnello has joined #openstack-infra13:13
*** baoli has quit IRC13:14
Clintkrotscheck1: get him13:14
*** nadya has joined #openstack-infra13:14
*** nadya is now known as Guest1382613:15
*** amotoki has quit IRC13:15
*** xyang1 has joined #openstack-infra13:16
*** davideagnello has quit IRC13:18
dhellmannAJaeger: now that we have content on it would be good to get the redirect in place so folks don't find the stale content at the old location:
*** krotscheck has joined #openstack-infra13:18
openstackgerritZara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events
*** links has quit IRC13:20
*** sdake has joined #openstack-infra13:20
jprquestion: if a jenkins job has a node restriction on it, does that require ZUUL_NODE to be set by parameter-function to the matching value so that jenkins can match the job description correctly?13:21
*** claudiub has quit IRC13:21
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud
*** rfolco has joined #openstack-infra13:23
*** hashar has quit IRC13:24
krotscheck1sdague, dstufft, clarkb: Could not fetch URL 404 Client Error: Not Found - skipping13:24
krotscheck1Case sensitive.13:24
dstufftyou need to normalize13:24
dstufftthe full normalization rules are in PEP 503 fwiw13:25
* krotscheck1 ponders the best place to do that.13:25
openstackgerritZara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events
*** yamamoto has quit IRC13:27
*** abregman has joined #openstack-infra13:29
*** Jeffrey4l has joined #openstack-infra13:31
*** edmondsw has joined #openstack-infra13:32
*** sdake_ has joined #openstack-infra13:33
*** flepied1 has quit IRC13:33
AJaegerthanks, sdague13:35
AJaegerdhellmann: Let me review...13:35
*** dtantsur is now known as dtantsur|brb13:35
*** sdake has quit IRC13:36
*** jcoufal has joined #openstack-infra13:36
AJaegerkrotscheck1: you can get runtimes out of grafana like done here - scroll to the bottom13:37
*** tongli has joined #openstack-infra13:38
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Don't add non-existent items when resolving lists
krotscheck1AJaeger: Thanks :)13:38
*** trown has joined #openstack-infra13:40
krotscheck1AJaeger: Hrm. Good chunk of reduction, but not as impressive as I'd hoped.13:41
mordredkrotscheck1: it's still totally solid13:41
krotscheck1mordred: True. There's a noticable drop in runtime.13:42
mordredyah- and spread across all the builds, it should add up to additional daily node capacity13:43
anteayamordred: morning, I find antique stores to be much more fun that large box stores but yay for having space13:43
mordredanteaya: yes indeed13:43
*** jcoufal has quit IRC13:44
mordredanteaya: I typically do not enjoy large box stores - but if one must large box store, then empty is better than full :)13:44
anteayaso nice work on the timing of that13:44
*** flepied has joined #openstack-infra13:46
*** trown has quit IRC13:47
*** trown has joined #openstack-infra13:48
*** krotscheck1 has quit IRC13:48
AJaegeryes, thanks to everybody for that!13:50
anteayamorning AJaeger13:50
anteayaAJaeger: have a list?13:50
anteayas/have a list/have you a list13:51
AJaegeranteaya: a long one for jeblair ;) to discuss the renaming ones.13:51
AJaegeranteaya: thanks for asking. Could you look at and, please?13:51
AJaegerand this one, please
anteayaokey dokey13:52
*** weshay_xchat has quit IRC13:52
*** nikil89_ has quit IRC13:52
AJaegermordred: should I +2 your multiverse removal from project-config? Change
*** baoli has joined #openstack-infra13:53
*** baoli has quit IRC13:53
AJaegerwe didn't +A yet in case you wanted to babysit it...13:53
*** davideagnello has joined #openstack-infra13:54
AJaegermordred: should we push the apt mirror update into today's nodepool build? Then let's ask anteaya for a second +2A..
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure
*** baoli has joined #openstack-infra13:55
krotscheckmordred: ^^ don't merge that just yet, I have to get a legacy script ready to go.13:56
AJaegerkrotscheck: mark it as WIP for now so that it doesn't merge...13:56
*** devananda has joined #openstack-infra13:56
dimsAJaeger : anteaya : please add this to your review queue - - the nova py27/py34 with oslo.* master works well. want to run other projects against oslo.* master now.13:57
*** sdake_ has quit IRC13:57
*** sdake has joined #openstack-infra13:58
*** weshay_xchat has joined #openstack-infra13:58
*** weshay_xchat is now known as weshay13:58
*** davideagnello has quit IRC13:58
*** yamamoto has joined #openstack-infra13:58
AJaegerdims: I'd like to have sdague for nova and one of the neutron liasons like dougwig, mestery, or armax review it as well first - and best also other folks for their projects. This is going into the integrated gate at one point, so let's get consensus early13:59
*** sdake has quit IRC13:59
*** sdake has joined #openstack-infra14:00
*** baoli has quit IRC14:00
*** kushal has quit IRC14:00
anteayaAJaeger: mordred I have +2'd 277114 but am not comfortable +A'ing it as I feel like a mirror tourist rather than driver, would prefer if a mirror driver +A'd14:00
*** annegentle has joined #openstack-infra14:01
anteayadims: what AJaeger said14:01
*** baoli has joined #openstack-infra14:01
AJaegeranteaya: yeah, let's a mirror driver give his ok - and if mordred self-approves it's fine with me...14:01
*** jsavak has joined #openstack-infra14:01
*** abregman has quit IRC14:01
AJaegerdims: reviewing again, I have one concern: Once you put this into the gate, all the repos will be in the integrated gate - and thus you increased the number of repos in it ;(14:02
mordredkrotscheck: you have shown me shell things that are above my meager understanding14:02
AJaegerdims: Discuss this with integrated gate experts like sdague and mtreinish as well...14:02
mordredkrotscheck: what does <<< do?14:02
*** tiswanso has joined #openstack-infra14:02
*** rossella_s has quit IRC14:02
AJaegeranteaya: thanks for all the reviews!14:03
openstackgerritMerged openstack-infra/project-config: Rename stackforge-specs-jobs to build-only-specs-jobs
anteayaAJaeger: welcome14:03
*** rossella_s has joined #openstack-infra14:03
openstackgerritMerged openstack-infra/project-config: Re-enable ironic-ui translation
* anteaya goes back to skimming the email list14:04
*** gsagie has quit IRC14:05
*** tiswanso has quit IRC14:05
openstackgerritMerged openstack-infra/project-config: Do not publish openstackdocstheme documents
*** tiswanso has joined #openstack-infra14:06
mordredAJaeger: let's give the mailing list post a little more time to uncover any hostility14:06
*** jsavak has quit IRC14:07
*** kushal has joined #openstack-infra14:08
*** Jeffrey4l has quit IRC14:08
anteayajaypipes: nice work crafting a lovely gender neutral mailing list post, I appreciate it14:08
krotscheckmordred: It works?14:09
* krotscheck may have stackoverflowed that one.14:09
AJaegermordred: works for me...14:09
anteayajaypipes: I refer to
prometheanfireianw: does the finalize stage happen before the sytem is deployed to the target (raw/qcow)?14:09
*** YorikSar has quit IRC14:09
*** notnownikki has joined #openstack-infra14:09
mordredkrotscheck: :)14:10
*** jsavak has joined #openstack-infra14:10
*** YorikSar has joined #openstack-infra14:10
*** jaosorior has joined #openstack-infra14:11
dimsanteaya : AJaeger : how do i set it up so it's not in integrated-gate? all i need is periodic jobs that run every day14:11
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure
*** naresht has quit IRC14:11
*** rlandy has joined #openstack-infra14:11
*** dtantsur|brb is now known as dtantsur14:12
AJaegerdims: if it will be periodic, no worries...14:12
AJaegeronly if they share the gate14:13
dimsAJaeger : yep. i don't need this to run on check or gate...or vote14:13
anteayaAJaeger: my sense from the jobs that dims is creating is that they are designed to be periodic only14:13
anteayaAJaeger: it is a good concern to raise and I'm glad we are talking about it14:14
dimsanteaya : yes, the only reason i added them to experimental was to make sure i can test it14:14
dimsanteaya : AJaeger : once i kick the tires, we can drop that14:14
anteayabut as you point out the integrated bits affect the gate queue only14:14
*** jsavak has quit IRC14:14
anteayadims: drop experimental you mean14:14
AJaegeryes, I stand corrected - so should be fine and probably won't need buy-in - but let's still give a heads up to nova and neutron teams - especially neutron asks for review by their liasons14:14
*** jsavak has joined #openstack-infra14:14
dimsanteaya : yes14:14
anteayadims: great14:14
*** abregman has joined #openstack-infra14:14
anteayaAJaeger: yup, buy in is good, glad we had a moment to discuss it14:15
anteayadims: can you find folks in the other projects to review?14:15
AJaegerdims: I'll gave a +2 but added a comment  as well...14:16
*** Guest13826 has quit IRC14:16
mordredjeblair, fungi: when you awaken, I could use a +2 on and its immediate ancestor14:16
dimsanteaya : this is mainly for oslo team to make sure our code does not break other projects. i am running stuff on travis to do that right now ( and would like to get rid of my travis setup14:17
mordredI have performed the file moves locally14:17
dimsAJaeger : who is the neutron liaison? i can ping14:17
*** nadya has joined #openstack-infra14:18
anteayadims: armax or dougwig are good14:18
dimsthanks anteaya14:18
anteayathank you14:18
*** nadya is now known as Guest5691014:18
anteayadims: I'm not saying i think they will have an issue with what you propose14:19
anteayabut I think we are so large that many folks have things pop up that affect them that then never even knew about14:19
*** annegentle has quit IRC14:19
anteayaso a courtesty heads up goes a long way to reducing stress, I find14:19
*** annegentle has joined #openstack-infra14:20
*** jaosorior has quit IRC14:21
*** davideagnello has joined #openstack-infra14:21
*** jaosorior has joined #openstack-infra14:21
*** vsaienko has quit IRC14:21
dimsanteaya : good point14:22
mordreddims: also, I support getting your stuff off of travis :)14:22
dimsthanks mordred :)14:22
*** hashar has joined #openstack-infra14:23
anteayayeah that too14:23
*** kgiusti has joined #openstack-infra14:24
mordredjeblair: also - when you awaken, I'm confused by some permissions - putting the lists and db files into AFS and then running reprepro I get Error 13 creating lock file '/afs/': Permission denied!14:24
mordredjeblair: however, /afs/ has the same permissions as the other dirs14:25
mordredjeblair: (using fs la of course)14:25
*** davideagnello has quit IRC14:26
AJaegerI blogged about adding jobs to our CI at and will do a followup about templates. If you find anything there that is a good idea to add to the infra-manual, feel free to propose a change for it or tell me exactly what to propose...14:26
*** sabeen has joined #openstack-infra14:26
dimsanteaya : AJaeger : i think i am missing some bit as i don't see entries for periodic-nova-py27-with-oslo-master in
dimsanteaya : AJaeger : i was able to run that as experimental on a nova review, just don't see the daily run that was supposed to happen14:27
*** yamamoto has quit IRC14:28
AJaegerdims: found th eproblem: branch: master is a no-go14:28
AJaegerperiodic jobs cannot have a branch selection, send a patch to remove that please14:29
AJaegerYou have to tell your job that it's on master...14:29
AJaegertime for home made waffels, I'll be back later...14:29
dimsAJaeger : ack14:30
*** sabeen2 has joined #openstack-infra14:30
*** vincentfrancoise has joined #openstack-infra14:31
*** adrienverge has quit IRC14:31
*** esker has quit IRC14:31
openstackgerritGiulio Fidente proposed openstack-infra/project-config: Rename tripleo ceph job into upgrades
*** regXboi has joined #openstack-infra14:33
*** sabeen has quit IRC14:33
anteayaAJaeger: mmmmm waffels14:35
*** jsavak has quit IRC14:35
vincentfrancoiseHi everyone! I have a patchset (see that is waiting for reviews since last week. So if you could review it, that would be much appreciated14:35
*** yamahata has quit IRC14:35
*** jsavak has joined #openstack-infra14:36
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Drop branch selection on oslo-master periodic job
*** sc68cal has joined #openstack-infra14:38
*** salv-orlando has quit IRC14:41
anteayavincentfrancoise: one way of helping us reduce our review queue and get to your patch is to start reviewing patches yourself, reading this may be of interest to you:
*** ccrouch has joined #openstack-infra14:44
fungimordred: a little searching suggests that some kinds of file locking get weird on afs (as they do on nfs and other network filesystems), so i wonder if it's some other problem with the lock surfacing as a permissions issue?14:45
*** superdan is now known as dansmith14:45
*** adrienverge has joined #openstack-infra14:47
krotscheckWell, piffle.14:47
krotscheckfungi, mordred: It looks like my naïve "Get distribution name by dash-split" method resulted in some of our packages being misfiled in the wheel mirror.14:47
openstackgerritMerged openstack-infra/system-config: Update reprepro options file with valid options
mordredfungi: ah. duh14:47
mordredfungi: so maybe keeping the db files in AFS is a bad idea and we should just store them locally on the mirror-update host14:48
krotscheckUPside: We can squeeze more performance out of wheel!14:48
krotscheckDownside: We kinda have to resort everything.14:48
pabelangerclarkb: greghaynes: Ya, I plan to look into fedora / centos mirrors this week. Mostly to see what AFS is all about.  I don't think anybody else is working on it, but correct me if I am wrong14:48
*** davideagnello has joined #openstack-infra14:48
mordredpabelanger: I was going to work on it if nobody else did - so I'm happy to work with you on it when you get there14:49
fungimordred: or maybe keeping the lockfile on the builder is sufficient (we don't expect to have additional mirror builders attempting concurrent access to the reprepro db, right?)14:49
fungibut still, occam says we're probably just missing something permissions wise14:49
mordredfungi: there does not seem to be a way to configure lockfile location separate from db location14:50
*** nelsnelson has quit IRC14:50
vincentfrancoiseanteaya: ok, I'll do as you suggest14:50
pabelangermordred: Sure. Give me a few hours to clean up some other stuff.  I still need to try out the afs client14:51
mordredpabelanger: no rush - still finishing off apt mirrors14:51
*** EricGonczer_ has joined #openstack-infra14:51
fungimordred: looks possibly related14:51
jeblairwhat does it try to do with the lockfile?14:52
fungino, nevermind that's error 11 not 1314:52
pabelangermordred: great. Ya, I'll be sure to see how that all works first14:52
*** sridhar_ram has joined #openstack-infra14:52
*** sdake has quit IRC14:52
*** annegentle has quit IRC14:53
jeblairmordred: can you help me run it in screen to get the error?14:53
*** davideagnello has quit IRC14:53
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update for the upgrades job
jeblairmordred: i have rejoined your screen session14:53
mordredjeblair: neat!14:54
mordredjeblair: one sec14:54
jeblair(oh wow, that still hasn't landed :)14:54
*** salv-orlando has joined #openstack-infra14:55
mordredk. seems to just be a chmod needed on the dir :)14:55
mordredjeblair: so let's land those changes real quick14:55
*** amitgandhinz has joined #openstack-infra14:55
fungiwhich afs-related changes did i miss?14:55
krotscheckjeblair: THere's a thing in backscroll that's a little disconcerting.14:55
anteayavincentfrancoise: thank you, do ask if you have any questions as you learn how to review14:55
fungithought i had reviewed most of them14:55
*** abregman has quit IRC14:55
mordredjeblair. fungi: first one just landed 7 minutes ago14:56
fungia few of them in the series you mentioned earlier still need an additional +2 besides me14:56
fungiso those i guess14:56
*** esikachev has quit IRC14:57
fungii can switch my -1 on 277189 to +2 if you want to tackle my comments in a separate change14:57
fungiwhat you have will technically work, just could be better for the long-term14:57
vincentfrancoiseanteaya: ok will do14:58
mordredfungi: I could go either way - whatever  you prefer14:58
mordredfungi: however, I'm wondering if maybe I should not import any keys for debian until we decide to run a debian mirror14:58
mordredsince debian has many keys14:58
krotscheckWait! I am wrong.14:58
anteayavincentfrancoise: thank you14:58
krotscheckIt's a little odd though. Python package names have to be valid identifiers, meaning they can't contain dashes, but the normalized pep503 repository folder name requires the use of dashes.14:59
fungimordred: i agree, how about strip that down to just the ubuntu key and switch it to a full fingerprint and i'll +214:59
*** vgridnev has quit IRC14:59
*** esikachev has joined #openstack-infra14:59
jeblairkrotscheck, sdague: job runtimes are _definitely_ in graphite15:00
openstackgerritMonty Taylor proposed openstack-infra/system-config: Import gpg keys for mirroring into local keyring
mordredfungi: how about that15:00
* anteaya is very depressed as she has to set a mouse trap :(15:00
jeblairkrotscheck, sdague: also, i have seen devstack-reported pip_install times go from ~20 minutes to 2.15:00
*** krtaylor has joined #openstack-infra15:01
fungimordred: lgtm15:01
sdaguejeblair: yeh, there is definitely an impact when I'm looking at spot timings15:01
*** sigmavirus24_awa is now known as sigmavirus2415:01
fungianteaya: they make non-lethal mouse traps if you'd rather catch and release, but you have to take them pretty far if you don't want them finding their way back15:02
*** nmagnezi has quit IRC15:02
jeblairsdague: when SpamapS finishes his work, we may be able to easily dump pip_install, etc, timings into graphite15:02
sdaguejeblair: yeh, I think that was an intent by mtreinish / SpamapS15:02
*** egonzalez has joined #openstack-infra15:03
jeblairsdague: ah cool; i knew about the actual openstack metrics, didn't know the devstack phases had been thought of15:03
*** piet has joined #openstack-infra15:03
jeblairmordred: i guess the debian key id was wrong in earlier versions of 2B90D010 ?15:04
openstackgerritMerged openstack-infra/storyboard: Don't add non-existent items when resolving lists
eantyshevanteaya, jeblair: May I ask you again to review , please? This fixes some broken Zuul functionality15:05
jeblairmordred: why move the db and lists to afs?15:05
*** tdurakov has joined #openstack-infra15:06
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job and ceph in the nonha
anteayafungi: thanks, yeah it would just end up in my basement again :(15:09
*** jaypipes has quit IRC15:09
*** arxcruz has quit IRC15:09
anteayafungi: mostly I'm at odds with my cat who is wonderful at catching mice outside, apparently inside doesn't interest her15:09
jeblairkrotscheck: lgtm15:09
krotscheckjeblair: It's broken15:09
jeblairkrotscheck: still lgtm.  :)15:10
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha and upgrdes job
zxiiroelectrofelix: Have you seen my latest comments regarding the mapping patch?15:10
krotscheckjeblair: I'm replacing all special characters in package names _before_ splitting on '-'15:10
krotscheckjeblair: WHich means that xstatic_foo_bar turns into xstatic-foo-bar and the script things the name is xstatic.15:10
krotscheckMind you, having all xstatic packages be undiscoverable is something that I actually would condone.15:11
krotscheckBut, well, the sideffect isnt' so hot15:11
krotscheckI'm testing a fix right now.15:11
zxiiroelectrofelix: actually I think I'll make a mailing list post as this is an important topic in my opinion. We are being hit buy this use case recently.15:11
jeblairkrotscheck: gotcha15:11
jeblairmordred, fungi: are we ready for ?15:12
*** ajmiller has joined #openstack-infra15:13
*** eharney has joined #openstack-infra15:13
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure
krotscheckjeblair: ^^ is the correct version.15:13
openstackgerritMerged openstack-infra/system-config: Update paths for AFS mirror
krotscheckjeblair: That'll require some cleanup for the mirrors. Do you think it'll be safe to flat-out delete any wheels that don't have normalized folder names?15:14
krotscheckOr should I copy them to their new location.15:14
zigofungi: Hi. Could I ask you to review (debian image patch) ? It's stalled with nobody reviewing... :(15:15
jeblairkrotscheck: delete should be fine -- the first mirror run with that change will put them in place.15:15
*** annegentle has joined #openstack-infra15:15
*** ildikov has quit IRC15:16
*** zeih_ has quit IRC15:17
jeblaireantyshev: lgtm15:18
*** rbrndt_ has joined #openstack-infra15:18
eantyshevjeblair: any further steps from me expected?15:18
jeblaireantyshev: no, thanks! it's approved (but let me know if it doesn't merge)15:19
eantyshevjeblair: thanks!15:19
mordredjeblair: I thnk I'd like for the cron to be running and us happy with it before throwing the configure-mirror switch15:21
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard
jeblairmordred: ok15:21
mordredjeblair: as for why move them to AFS - it's either AFS or local /var ... but if we move them to local var and then mirror-update goes away, we'll have to rebuild the mirror from scratch15:22
jeblairmordred: got it15:22
*** arxcruz has joined #openstack-infra15:22
*** esikachev has quit IRC15:22
*** vgridnev has joined #openstack-infra15:23
*** nelsnelson has joined #openstack-infra15:24
mordredkrotscheck: it's safe to land that script patch yeah? or do we need to coordinate the cleanup?15:24
*** amotoki has joined #openstack-infra15:24
krotscheckmordred: Landing it won't break anything.15:24
mordred+a then15:24
krotscheckmordred: woot15:25
openstackgerritIgor Belikov proposed openstack/diskimage-builder: Install additional packages in debian-minimal
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add ZFSonLinux experimental Tempest job for Manila
*** rossella_ has joined #openstack-infra15:26
mordredfungi: 2016-02-08 15:14:01.516 | Error: Parameter key_id failed on Gnupg_key[Ubuntu Archive]: Invalid key id 0x630239CC130E1A7FD81A27B140976EAF437D05B5 at /home/jenkins/workspace/gate-infra-puppet-apply-bare-trusty/openstack-infra/system-config/modules/openstack_project/manifests/mirror_update.pp:10115:26
AJaegerjeblair: I reworked the timeout changes as suggested: First upgrade timout to 10 mins for merged change: 277155, then update d-g to 10mins: 277154 - and updated the three remaining timeout changes to include the increase: 276859, 277061, 277069. Could you review these laster, please?15:27
*** haypo has left #openstack-infra15:27
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add ZFSonLinux experimental Tempest job for Manila
jeblairAJaeger: will do asap15:27
*** rossella_s has quit IRC15:28
fungimordred: try dropping the 0x i guess15:28
mordredfungi: seems it only accepts 8 or 16 character IDs15:28
*** egonzalez has quit IRC15:28
* mordred reading source15:28
fungiwow, that's terrible15:28
fungialso i got to thinking, it's sort of silly having a puppet module retrieve keys from a keyserver when you already know the key you want. does that module allow us to just supply the public key?15:29
jeblairif not terrible, it's extremely unlikely to be great.15:29
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool
*** davideagnello has joined #openstack-infra15:29
mordredfungi: k. it seems like it's just doing passthrough: gpg --fingerprint --with-colons #{resource[:key_id]}15:29
mordredfungi: so removing the 0x may work15:29
mordredfungi: also, yes, it does allow passing in content15:30
fungiyeah, try dropping the 0x for now i guess. i tend to use it to be explicit, but that may be a gpg2-onlyism15:30
krotscheckjeblair, mordred, fungi: will remove all the packages that don't match the normalized names. The impact of this is minimal, since the jenkins slaves can't find those packages anyway.15:30
krotscheckWe'll want to run that after merges15:31
jeblairkrotscheck: ++  fungi do you want to exercise your afs perms, or should i run that?15:31
*** egonzalez has joined #openstack-infra15:31
fungijeblair: i'm in a meeting for the next half hour, but i can try it after i'm done15:31
jeblairfungi: we need to wait a bit anyway15:31
*** zz_dimtruck is now known as dimtruck15:31
fungiokay, i'll take a shot at it in ~3015:32
krotschecknote, it won't delete summary directories that have lowercase names. But _shrug_.15:32
krotscheckThose'll come back as soon as those packages are rebuilt15:32
krotscheckSo I didn't see much point to deleting them.15:32
*** hdd has joined #openstack-infra15:33
*** achanda has joined #openstack-infra15:33
*** davideagnello has quit IRC15:34
mordredfungi: it's a bug in the puppet module:
*** jaypipes has joined #openstack-infra15:34
mordredfungi: overeager validation code15:34
*** annegentle has quit IRC15:35
*** annegentle has joined #openstack-infra15:36
mordredfungi: we've actually already imported the key on the server, so this is brace and suspenders anyway - how about I put in the 16-char version, then submit a patch upstream to accept the longer id15:36
*** claudiub has joined #openstack-infra15:36
fungimordred: yeah, i guess for now put in whatever works, and later we can consider just puppeting in the key material instead (so that our puppet isn't dependent on being able to connect to the keyserver)15:37
*** mtanino has joined #openstack-infra15:37
fungithat also solves the hkps:// dilemma, because we can retrieve the key manually via an authenticated protocol and then just puppet the blob15:38
*** esker has joined #openstack-infra15:38
*** esker has quit IRC15:39
jeblairyeah, in my experience, depending on keyservers is :(15:39
fungi(public service announcement: normal pgp keyserver protocol does not validate the connection, which makes it hard to bootstrap a web of trust)15:39
*** esker has joined #openstack-infra15:39
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool
krotscheckWait, we already have afs mirros?15:40
*** jsavak has quit IRC15:40
krotscheckSorry, I meant APT mirrors.15:40
krotscheckThat was fast.15:40
jeblairkrotscheck: yep!15:40
fungikrotscheck: in fairness, mordred had most of it already written in various states of decay15:41
openstackgerritMonty Taylor proposed openstack-infra/system-config: Import gpg keys for mirroring into local keyring
mordredfungi, jeblair: updated with 16-char for now15:41
jeblairfungi: i think he only started writing it when we started on the afs pypi mirrors, but yeah, not *strictly* linear15:41
*** vincentfrancoise has left #openstack-infra15:41
krotscheckjeblair: Neat! How much of a job improvement have we seen?15:41
jeblairkrotscheck: oh, not in prod yet15:42
*** rcernin has quit IRC15:42
mordredit was easy once krotscheck and jeblair had done all the hard bits15:42
anteayaif an infra-root has a moment, jenkins06 doesn't seem to be part of the team effort currently:
*** ddecapit has joined #openstack-infra15:42
jeblairkrotscheck: they exist, but we haven't flipped the configure-mirror switch yet15:42
*** ddecapit is now known as DuaneDeC715:42
*** doug-fish has joined #openstack-infra15:42
fungimordred: oh, i thought you already had some earlier patches for setting up deb mirrors15:42
mordredkrotscheck: you can look at the run
openstackgerritMerged openstack-infra/project-config: Enable multinode testing in Watcher
mordredkrotscheck: to see devstack running against them15:42
krotscheckI was just doing that15:43
jeblairkrotscheck: i'm not expecting a noticeably improvement in runtime (maybe i'll be surprised?), but we are expecting improvements in reliability.15:43
*** achanda has quit IRC15:43
mordredsame here15:43
fungibut yeah, it just seems especially silly to have configuration management retrieve a key from a keyserver via key id (if you have the key id you already know what key you need), especially when you don't need any other metadata about they key either (reprepro isn't validating other signatures of that key for example)15:43
*** tzn has quit IRC15:43
mordredfungi: now - there is another discussion we shoudl maybe have15:43
mordredfungi: which is "do we want to sign the repo we're publishing"15:43
AJaegerdhellmann: could you reveiw solum's reno addition again, please?
mordredfungi: my current stance is "no" because that makes these mirrors less desirable for people who are not our gate15:44
AJaegerdhellmann: the redirect to works...15:44
mordredfungi: however, once we have them, there's no good reason to not also use them for our long-lived servers ... at which point I think having a signed repo wouldn't be terrible15:44
fungimordred: i guess that depends on how much we want to trust the jobs we're running on workers relying on them, and whether we want to use those mirrors for our other servers too15:45
krotscheckSo. Many. Mirrors15:45
fungiyeah, that15:45
fungimordred: i was honestly surprised you could turn secure-apt off and ignore the lack of signing15:46
*** amotoki has quit IRC15:46
fungiit's been default for so long i just assumed the old mode had been disabled years ago15:46
jeblairfungi, mordred: as they stand -- i don't think we need to use these for our own servers.  when we start dropping our own packages in them, that changes my answer.  :)15:46
mordredjeblair: yah. and those would be more interesting signatures too15:47
*** Sukhdev has joined #openstack-infra15:47
jeblairmordred: we could make that a separate archive though, right?15:47
mordredjeblair: yes. we could make that a separate archive15:47
mordredjeblair: although we could also make it just a separate component in this archive15:48
mordredjeblair: so we'd have trusty trusty-updates trusty-backports trusty-infra for instance15:48
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Revert "Enable voting for Puppet OpenStack Integration jobs"
*** jsavak has joined #openstack-infra15:48
EmilienMdmsimard: ^15:48
mordredjeblair: haven't thought through that fully yet since it's not really an urgent for us15:48
openstackgerritMerged openstack-infra/project-config: Normalize wheel folder structure
*** ildikov has joined #openstack-infra15:49
EmilienMI have an outstanding request for infra, puppet group would like to disable voting for some jobs, just for today & maybe tomorrow. Please review
*** links has joined #openstack-infra15:49
jeblairmordred: ya15:49
*** mikelk has joined #openstack-infra15:50
dhellmannAJaeger : thanks!15:50
dhellmannAJaeger : looking at the solum change now15:50
*** tzn has joined #openstack-infra15:52
*** egonzalez has quit IRC15:52
*** egonzalez has joined #openstack-infra15:53
mordredjeblair: hrm. I got the lockfile permission error again15:53
*** tzn has quit IRC15:53
*** aspiers has quit IRC15:53
*** tzn has joined #openstack-infra15:53
*** pradk has joined #openstack-infra15:54
*** jsavak has quit IRC15:54
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
*** Daisy has quit IRC15:56
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix for stable/liberty job
*** davideagnello has joined #openstack-infra15:56
fungimordred: and you've confirmed there's no existing file there, right?15:57
mordredShrews: ^^ you're going to want to move not copy15:57
anteayaEmilienM: I commented, you need to have some job running for your gate queue15:57
mordredShrews: since ~ overrides /etc15:57
*** links has quit IRC15:57
Shrewsmordred: if it had found ~, it wouldn't even be a problem  :)15:57
Shrewsmordred: different user (stack vs. jenkins)15:57
EmilienManteaya: that's a revert, it was like this before15:57
mordredShrews: isn't it the sed that's the issue though?15:57
*** esikachev has joined #openstack-infra15:58
Shrewsmordred: yes, b/c it couldn't find the file in /etc15:58
EmilienManteaya: see
*** aysyd has quit IRC15:58
mordredShrews: right. but if you copy a file to etc and then sed it - the tests are going to still read the one in ~ and then they'll be running against the wrong keystone api version15:58
*** jsavak has joined #openstack-infra15:58
*** sridhar_ram has quit IRC15:58
Shrewsmordred: the tests can't SEE the one in ~15:59
*** aysyd has joined #openstack-infra15:59
mordredShrews: 0_o15:59
anteayasc68cal: go go go15:59
EmilienManteaya: we have some jobs in gate15:59
*** vgridnev has quit IRC15:59
Shrewsmordred: b/c the tests run as the jenkins user, but the file is written to the stack user's ~15:59
EmilienManteaya: see for example, we have lint & syntax15:59
sc68calanteaya: hmm?15:59
openstackgerritMichael Johnson proposed openstack/requirements: Octavia requires fixes in taskflow 1.26.0 to function
mordredShrews: k.15:59
anteayasc68cal: the mailing list post, go you15:59
sc68calanteaya: ah. I thought so. Thanks :)16:00
*** jecarey has joined #openstack-infra16:00
*** EricGonc_ has joined #openstack-infra16:00
anteayasc68cal: :)16:00
*** harlowja_at_home has joined #openstack-infra16:00
anteayaEmilienM: I'm missing how lint and syntax are running on that repo, however yes, those jobs would be sufficient to have something running in the gate to merge patches16:01
*** davideagnello has quit IRC16:01
Shrewsmordred: $BASE/new is not ~jenkins ... if that makes sense16:01
anteayayou must have some meta template or something16:01
EmilienManteaya: let me show you the code, a sec16:01
*** esikachev has quit IRC16:02
AJaegeranteaya: the jobs use more than one template16:02
*** infra-red has quit IRC16:02
EmilienMand anyway, it's a revert16:03
*** vgridnev has joined #openstack-infra16:03
*** EricGonczer_ has quit IRC16:03
*** markmcd has quit IRC16:03
*** markmcd has joined #openstack-infra16:04
anteayaEmilienM: no I get it, you are changing the template16:04
*** trown|brb has joined #openstack-infra16:04
*** markmcd1 has joined #openstack-infra16:04
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo
anteayapuppet has a lot of templates16:04
anteayaI thought you were removing the jobs from a repo16:04
*** ajmiller_ has joined #openstack-infra16:05
EmilienManteaya: thanks! no worries16:05
anteayasorry I was confused16:06
*** placko has quit IRC16:06
*** esikachev has joined #openstack-infra16:07
EmilienManteaya: it's monday :-)16:07
anteayaand I'm just confused these days16:08
anteayamore like it is me16:08
anteayaanyway, hope you get things sorted out for yourself16:08
openstackgerritJames E. Blair proposed openstack/gertty: Add debug-sync option
openstackgerritJames E. Blair proposed openstack/gertty: Add support for last_seen
openstackgerritJames E. Blair proposed openstack/gertty: Add support for customized dashboard sorting
openstackgerritJames E. Blair proposed openstack/gertty: Make sorting commands use two two keys
openstackgerritJames E. Blair proposed openstack/gertty: Fix return values in handleCommands in project_list
pabelangerAJaeger: looking at the debian jessie patch, do we actually use the Restricted repo from Ubuntu? Would be interested what happends if that was removed16:09
jeblairfungi, mordred: ^ the tip of this stack has support for a new dashboard that shows you the changes you have most-recently seen16:09
* Shrews wonders what jeblair means by "two two keys"16:09
*** ajmiller has quit IRC16:09
jeblairShrews: ha!16:09
AJaegerpabelanger: mordred proposed a patch to remove it from Ubuntu - see also openstack-dev mailing list. Tests look very good!16:09
pabelangerAJaeger: Oh, nice16:09
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch non-infra Ansible roles to using 'linters' tox target
odyssey4mepabelanger note:
*** trown has quit IRC16:10
jeblairfungi, mordred, Shrews: so when you saw earlier but need to come back to, it's easy to find it.16:10
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo
jeblairfungi, mordred, Shrews: so when you have a patch you saw earlier but need to come back to, it's easy to find it.16:11
fungithat sounds very useful16:11
fungiwill try to find time to test it out16:11
jeblairfungi, mordred, Shrews: it has a schema change, so making a db backup may be a good idea16:12
*** trown|brb has quit IRC16:12
*** trown has joined #openstack-infra16:13
jeblairfungi, mordred, Shrews: that series also includes conflicts-with support (which is also a schema change)16:14
Shrewsooh! now that i _really_ like16:14
*** isaacb has quit IRC16:15
openstackgerritZara proposed openstack-infra/storyboard: Add column for task links
fungijeblair: krotscheck: for i need to do that on the r/w volume obviously, should i be trying to set my workstation up as an openafs (i haven't gotten around to that yet) client and do the deletes with my admin credentials, or just punt and do it on one of our systems that are already set up?16:15
jeblairfungi: shouldn't be much of a difference (the operations are lightweight, so being in the same DC won't be too much of an advantage).  so whichever you want.  :)16:16
*** Guest56910 has quit IRC16:16
jeblairfungi: but you can use your admin creds either way16:16
openstackgerritMerged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 2]
fungifor the sake of personal interest, i'll give the client setup a shot. are the steps at still accurate?16:17
*** DuaneDeC7 has quit IRC16:18
mordredfungi: yes16:19
*** DuaneDeC7 has joined #openstack-infra16:19
Shrewsjeblair: if running from source, do i need to 'pip install -e .' again to get the schema change?16:19
*** armax has joined #openstack-infra16:20
jeblairShrews: if you last installed with 'pip install -e .' you don't need to do it again; if you did something else previously, then yes.16:20
*** aspiers has joined #openstack-infra16:21
*** [HeOS] has joined #openstack-infra16:21
*** HeOS has quit IRC16:22
mordredjeblair: oh - did you see my earlier comment about having the lockfile problem again?16:22
mordredjeblair: it's in teh scrollback in teh screen16:23
*** vsaienko has joined #openstack-infra16:24
*** sbalukoff has quit IRC16:24
vsaienkoHello infra team, could you please have look
*** hashar has quit IRC16:25
*** tdurakov has quit IRC16:25
devananda^ is a small g-r update to add netmiko, which is a pre-requirement for testing neutron/ironic integration in in our gate16:26
*** tdurakov has joined #openstack-infra16:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch non-infra Ansible roles to using 'linters' tox target
openstackgerritZara proposed openstack-infra/storyboard: Add column for task links
*** Swami has joined #openstack-infra16:29
*** tcammann has joined #openstack-infra16:30
*** esp has joined #openstack-infra16:30
*** vgridnev has quit IRC16:31
jeblairmordred: oh, hrm, let me dig that up...16:31
*** vgridnev has joined #openstack-infra16:31
*** tcammann_ has joined #openstack-infra16:31
jeblairmordred: it looks like you ran it without tokens16:32
*** NobodyCa1 has joined #openstack-infra16:32
jeblairmordred: sometimes when i'm doing stuff like this, i 'k5start -- bash'16:32
mordredjeblair: I just ran the script though16:32
mordredjeblair: expecting the scripts k5starts to do the right thing16:33
jeblairmordred: oh, reprepro-mirror-update is the script, sorry16:33
mordredI see it16:33
mordredjeblair: thanks16:33
*** deva_ has joined #openstack-infra16:33
mordredjeblair: dumpunreferenced16:34
mordredisn't k5started16:34
jeblairah and that takes a lockfile?16:34
mordredbecause it does things with the db16:34
jeblairmordred: what about the next two?16:34
mordredprobably yes16:34
mordredpatch coming16:34
*** adrienverge has quit IRC16:35
*** adrienve1 has joined #openstack-infra16:35
*** tcammann has quit IRC16:36
*** tdurakov has quit IRC16:36
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use k5start even for the readonly reprepro actions
mordredjeblair: ^^16:36
*** tcammann_ has quit IRC16:36
*** tdurakov has joined #openstack-infra16:37
*** NobodyCa1 has quit IRC16:37
*** esp has quit IRC16:37
*** deva_ has quit IRC16:38
*** savihou has quit IRC16:38
fungimordred: any reason not to just fold k5start -t -f /etc/reprepro.keytab service/reprepro -- into $REPREPRO?16:38
fungimordred: also why is it --$REPREPRO now (should it be -- $REPREPRO there instead)?16:39
mordredfungi: good call on both16:39
*** abregman has joined #openstack-infra16:39
EmilienMoutstanding request, can we land ? it's blocking puppet CI, thanks16:40
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use k5start even for the readonly reprepro actions
*** salv-orl_ has joined #openstack-infra16:40
mordredfungi: how's that?16:40
*** kushal has quit IRC16:42
*** tdurakov has quit IRC16:42
*** salv-orlando has quit IRC16:43
*** markmcd has quit IRC16:43
*** markmcd1 has quit IRC16:43
*** egonzalez has quit IRC16:44
*** jecarey has quit IRC16:45
*** jgriffith_away is now known as jgriffith16:45
daemontoolfungi,  jeblair  here
daemontoolwe see the error>16:45
daemontool2016-02-08 14:50:04.603 | + find .tox -type f -name '*.log' -exec mv '{}' '{}.txt'16:45
daemontool2016-02-08 14:50:04.603 | find: missing argument to `-exec'16:45
daemontool2016-02-08 14:50:04.603 | + exit 116:45
*** markmcd has joined #openstack-infra16:45
daemontoolI think \; is missing there?16:45
*** esikachev has quit IRC16:46
*** armax has quit IRC16:46
fungidaemontool: yes, it looks like it16:46
daemontoolis that something that belongs to infra?16:46
daemontoolor us?16:46
*** tdurakov has joined #openstack-infra16:46
fungidaemontool: yep, i even think it's a bug introduced in a recentish patch of mine16:46
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Clarify Munch object usage in documentation
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Add docs tox target
daemontoolfungi,  ok ty let me know if there's anything we can do16:46
fungidaemontool: is the job failing, or is that just noise in the log?16:46
daemontoolthe job is failing16:46
fungioh, then we need to delete images16:47
Shrewsmordred: w00t. job fixed:
*** infra-red has joined #openstack-infra16:47
mordredjeblair, fungi: the version of the script in that patch works. it's running a vos release now, which I'm hoping will not be terribly long since we've done a few here already and there shouldn't be THAT many changes16:48
fungidaemontool: um, that job for the log you linked failed a unit test16:48
daemontoolfungi,  ah ok16:48
*** cloudtrainme has joined #openstack-infra16:48
*** gyee has joined #openstack-infra16:48
*** vivekd has joined #openstack-infra16:48
fungiare you sure that's not why the job is failing?16:48
daemontoolyes sorry  FAIL: tests.test_config.TestConfig.test_ini16:48
daemontoolnope I'm not sure, checking it16:49
*** kzaitsev_mb has quit IRC16:49
fungidaemontool: basically the find error you quoted would cause any job run via our script to fail16:49
fungiif it's not causing all jobs to fail, then we can treat the fix for the script as slightly less urgent16:49
fungijust embarrassing (on my part)16:49
*** infra-re_ has joined #openstack-infra16:49
clarkbfungi what is the purpose of moving tox.ini?16:50
*** hongbin has joined #openstack-infra16:50
fungiclarkb: where is it moving tox.ini?16:50
clarkboh I misread it is tox logs16:50
clarkbfungi my early  morning brain misparsed the find command that is failing16:50
*** fhubik is now known as fhubik_brb16:51
*** hongbin has left #openstack-infra16:52
*** infra-red has quit IRC16:52
*** DuaneDeC7 has quit IRC16:52
openstackgerritZara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events
*** shardy has joined #openstack-infra16:53
*** matrohon has quit IRC16:53
*** nadya has joined #openstack-infra16:54
*** nadya is now known as Guest5544716:54
*** ashleighfarnham has joined #openstack-infra16:54
*** mikelk has quit IRC16:54
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add missing semicolon in
anteayaclarkb: morning16:55
fungidaemontool: clarkb: AJaeger: sdague: ^16:55
mordredjeblair: ooh16:55
mordredjeblair: new fun error for you16:55
mordredjeblair: I ran the script once, it succeeded16:55
mordredjeblair: I ran it again to double check16:55
greghaynes might be of interest to you - we can now install pip using get-pip without ever installing it as a package first16:56
mordredjeblair: and got16:56
mordredjeblair: /afs/ initialization for service/reprepro@OPENSTACK.ORG not found, forgetting anyway16:56
mordredUnable to forget unknown filekey 'Kerberos initialization for service/reprepro@OPENSTACK.ORG'.16:56
*** harlowja_at_home has quit IRC16:56
*** jpr has quit IRC16:56
*** Daisy has joined #openstack-infra16:56
fungidaemontool: fwiw, that script doesn't run under errexit so that find syntax error is basically benign (it's no worse than things were before the patch which introduced it)16:57
*** xyang1 has quit IRC16:57
daemontoolfungi,  ok thanks for the follow up16:57
electrofelixzxiiro: few days behind email, so prehaps haven't seen it yet16:57
*** DuaneDeC7 has joined #openstack-infra16:58
*** tzn has quit IRC16:59
*** annegentle has quit IRC16:59
*** fhubik_brb is now known as fhubik16:59
*** annegentle has joined #openstack-infra17:00
*** reed_ has joined #openstack-infra17:00
*** jgriffith is now known as jgriffith_away17:00
*** jsavak has quit IRC17:01
mordredfungi: could I get you to +2 again?17:01
*** Daisy has quit IRC17:01
*** DuaneDeC7 has quit IRC17:02
notmorgansdague, dtroyer: for the health of ELK, dropping microseconds is important. see the comment on the review why we can't just use asctime from oslo_log [or tell me we don't care about apache 2.2 :) ]17:03
notmorganclarkb: ^ cc17:03
krotscheckfungi, jeblair, mordred: Are we ready to get rid of the old pypi mirrors?
*** markmcd has quit IRC17:03
*** kushal has joined #openstack-infra17:04
*** jgriffith_away is now known as jgriffith17:04
*** apoorvad has joined #openstack-infra17:04
sdaguenotmorgan: will that give us msec still?17:04
notmorgansdague: nope17:04
*** annegentle has quit IRC17:04
notmorgansdague: we get microseconds or seconds17:04
*** jsavak has joined #openstack-infra17:05
sdaguewe really need msec17:05
*** salv-orl_ has quit IRC17:05
notmorgansdague: because apache 2.2 has no ErrorLogFormat, so it always adds it's own timestamp17:05
notmorganthis means CentOS would be <timestamp> <timestamp> <log>17:05
*** salv-orlando has joined #openstack-infra17:05
sdagueoh, this just exposes on centos?17:05
notmorganrather than <timestamp> <log>17:05
sdagueso, if this is just an ELK fix, we can fix it in os-loganalyze instead17:06
*** hdd has quit IRC17:06
*** markmcd1 has joined #openstack-infra17:06
*** markmcd has joined #openstack-infra17:06
*** hdd has joined #openstack-infra17:06
clarkbwe can fix it in logstash too17:06
clarkbbut microseconds provide zero valur17:06
notmorganso apache can do "seconds" or "microseconds"17:06
sdagueclarkb: sure, but miliseconds are kind of important in debugging17:06
clarkband will be negative value if we have to fix them17:06
notmorganin ubuntu we can omit apache's timestamp17:07
clarkbthe alternative is to have keystone log timestamps right?17:07
notmorganon centos we can't17:07
*** baoli has quit IRC17:07
openstackgerritZara proposed openstack-infra/storyboard-webclient: Rename board and worklist controllers
notmorgancentos uses apache 2.2 so we have no control on errorlog format17:07
sdagueok, lets narrow the scope of things17:07
*** reed_ has quit IRC17:07
sdaguewhat is the bug, and which platforms does it expose on?17:07
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Set PS4 to provide additional debug info
notmorgansdague: no bug. just microseconds is a lot of uniques to index17:08
notmorgansdague: centos uses apache 2.2, always includes the aapche timestamp in the errorlog, where keystone's log is output17:08
sdaguecentos 7?17:08
*** ashleighfarnham has quit IRC17:08
fungii'm surprised centos 7 didn't go with apache 2.417:09
notmorgani couldn't find a standard apache 2.4 rpm17:09
notmorganit looks like EPEL might have it17:09
BobBallAnyone seen zuul not starting but not logging anything as to why it's not started either?17:09
jlvillalsdague: ^^^  A PS4 patch to devstack-gate too. I used your updated PS417:09
gtmanfredcentos7 has 2.4.917:09
notmorgangtmanfred: really? i was always getting 2.217:09
BobBallservice zuul start just completes and a zuul-server process exists for ~10 seconds then it's gone17:09
notmorganin centos717:09
*** jsavak has quit IRC17:09
gtmanfredone second17:09
*** cgalan_ has joined #openstack-infra17:09
notmorganif that is the case, easy we go back to oslo_log17:09
fungiapache 2.4 well predated centos/rhel 7 i'm pretty sure17:10
gtmanfredyeah, it should be 2.4 something17:10
anteayaBobBall: I haven't had that as a report, no17:10
anteayaBobBall: what version of zuul?17:10
*** jsavak has joined #openstack-infra17:10
gtmanfrednotmorgan: sorry, was 2.4.617:10
notmorgangtmanfred: ok then something was horked on my test17:10
notmorganbecause i kept getting apache 2.217:10
gtmanfredyeah, i would have had a fit if it was still 2.217:10
gtmanfredwell maybe not, cause i have moved all my stuff to nginx17:10
sdaguewe're definitely installing 2.4 in the gate17:10
notmorgansdague: ok i'll rework this to be asctime again in keystone instead17:11
AJaegerfungi, oops - thanks!17:11
*** infra-red has joined #openstack-infra17:11
notmorganand just assume devstack doesn't handle apache 2.217:11
BobBallanteaya: Not the most up to date, but we haven't updated recently and zuul has been working just fine until 3rd February after which it hasn't restarted and we can't restart it manually.17:11
fungiAJaeger: the original typo was entirely mine17:11
sdaguenotmorgan: I am totally ok with that assumption17:11
notmorgancan i drop the 2.2 ifblock as well around ErrorLogFormat?17:11
notmorgansdague: ok will clean a couple things up17:11
notmorgansdague: expect an update a little later today17:11
anteayaBobBall: :(17:11
sdaguenotmorgan: great17:11
*** esker has quit IRC17:11
anteayaBobBall: yeah, you are the first I have seen to report this experience17:12
funginotmorgan: i can't think of a platform we still strive to support which has apache 2.4 any longer17:12
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rename tag for making stable branches
notmorganfungi: wfm17:12
fungiso should be entirely safe to clean that up17:12
*** scheuran has quit IRC17:12
*** jlanoux has quit IRC17:12
anteayaBobBall: I have to eat soup but if you have more, I'll read it upon my return17:12
*** adrienve1 has quit IRC17:13
sdagueI think the only question is if there is anything running on stable that's borking logstash17:13
gtmanfrednotmorgan: if you want a newer version of 2.4, IUS packages the newest version for cent6 and 7
gtmanfredbut it sounds like you are just using stock packages17:13
*** dizquierdo has quit IRC17:13
*** fhubik has quit IRC17:13
sdaguebut, again, we're all trusty + centos7, so I think that seems fine to backport even17:13
*** samuelBartel has quit IRC17:13
sdaguegtmanfred: stock or what's in epel17:13
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/elastic-recheck: Add query for "Security group is still in use" bug 1543106
openstackbug 1543106 in tempest "TestRebuildInstanceWithVolume.test_rebuild_instance_with_volume failes with "Security group is still in use"" [Undecided,New]
*** dims has quit IRC17:14
sdaguebut, we seem to be getting 2.4.7 from what I can see, which is fine17:14
*** infra-re_ has quit IRC17:14
notmorgangtmanfred: nope any 2.4 works17:14
clarkbgreghaynes: for your comment on do you mean it isn't related to this change?17:14
*** asselin has joined #openstack-infra17:14
*** asselin__ has joined #openstack-infra17:15
* clarkb rechecks to see if it is super consistent17:15
greghaynesclarkb: I think there is a fail which caused a timeout, the fail in that comment I think is a second fail which is the known pymysql fixture issue that happens during cleanup17:15
*** mingyu has quit IRC17:15
jlvillalI have a minor patch to devstack-gate:  Has been 11 days without a review. So I thought I would ask for one :)17:15
*** ihrachys has quit IRC17:16
greghaynesclarkb: I have no idea why it timed out though17:16
*** tphummel has joined #openstack-infra17:16
*** mingyu has joined #openstack-infra17:16
*** mingyu has quit IRC17:16
*** mingyu has joined #openstack-infra17:16
*** xyang1 has joined #openstack-infra17:16
clarkbgreghaynes: gotcha17:18
clarkbalso no gearman timeouts that I can see in the server log, but we don't have images building this morning17:18
openstackgerritMerged openstack-infra/system-config: Import gpg keys for mirroring into local keyring
greghaynesclarkb: :(17:18
*** ccrouch has quit IRC17:18
openstackgerritZara proposed openstack-infra/storyboard: Add column for task links
jeblairkrotscheck: yes, i'm ready; fungi, clarkb: you okay dropping old pypi mirrors now?17:20
clarkbgreghaynes: doesn't look like the cron fired, so I think that implies the cron is still running17:21
fungikrotscheck: yeah, seems like it's time. i've seen/heard no failures related to the new mirrors17:21
*** cgalan_ has quit IRC17:21
clarkbI can do a thread dump and check17:21
clarkbjeblair: sure17:21
jeblairmordred: i got lost in scrollback, still seeing that error?17:21
*** yamamoto has joined #openstack-infra17:21
mordredfungi, jeblair: after comments from fungi and follow ups from cody-somerville and SpamapS - I believe I have made a mistake in the current organization of the AFS volume for APT17:22
clarkbhave we confirmed new images across the board using the new mirror?17:22
odyssey4mekrotscheck I'm told by dstufft that you've been looking into the normalising of names in the pip wheels? do you have a review up for a fix perhaps?17:22
mordredwhich means we need to rename and remount the current volume17:22
*** yamamoto has quit IRC17:22
mordredtl;dr - fungi is right, they cannot share a pool - so we want an apt.ubuntu and an apt.debian volume - each with their own reprepro setup17:22
greghaynesclarkb: huh, I wonder if it is hung up in waitForBuiltImages17:23
mordredshould be easy enough to change- but we should not roll the current thing live until we do17:23
jeblairclarkb: i think krotscheck did so a long time ago17:23
mordredjeblair: it's in the screen session17:23
krotscheckodyssey4me: Yes, it's merged.17:23
mordredjeblair: and I did not move forward on it because I do not understand the problem17:23
openstackgerritMerged openstack-infra/project-config: Revert "Enable voting for Puppet OpenStack Integration jobs"
greghaynesclarkb: Do you know about how long ago the last build was?17:23
fungithat is to say not roll the ubuntu mirror live and then try to rename things later when it comes time to add a debian mirror if we can avoid an outage by introducing some namespacing up front17:24
odyssey4mejeblair ah thanks, and thanks krotscheck for digging into it - any idea when a fresh wheel build will be done with that implemented?17:24
krotscheckodyssey4me: There'll be a bit of a delay as it shakes out and we have to deal with some of the cleanup.17:24
openstackgerritGábor Antal proposed openstack-infra/gearman-plugin: Removed unnecessary null check in StartJobWorker
jeblairmordred: re: move -- sounds good to me -- i guess we'll just stop manually running it, merge path changes, move volume, then start up again?17:24
jeblairmordred: re: error -- i'll dig17:24
mordredjeblair: yes17:24
fungikrotscheck: i'm working on the cleanup, though the sid machine i wanted to try it from was behind more than 1k package updates so i'm letting that finish first17:24
krotscheckodyssey4me: Either the next requrements merge, or the next periodic run, assuming that the script has already been synced to the wheel mirror.17:25
krotscheckodyssey4me: ^^ note fungi's comments on removing packages.17:25
krotscheckfungi: THanks :)17:25
*** tdurakov has quit IRC17:25
krotscheckodyssey4me: Out of curiosity, what's your interest?17:25
jeblairmordred: ha, i understand :)17:26
clarkbgreghaynes: 2016-02-07 15:06:57,122 ERROR nodepool.builder.BuildWorker: Exception while building image was last item logged by builder17:26
jeblairmordred: k5start output is being redirected to the cleanup list17:26
clarkbgreghaynes: I wonder if wait for built images expects that job to be restarted17:26
odyssey4meah, thanks fungi & krotscheck ... I'm working on a patch to make use of the wheels in the gate for OpenStack-Ansible:
clarkbgreghaynes: but since it isn't sits there waiting forever for an image build that failed?17:26
greghaynesclarkb: yep, so I need to fully page in how all it can get stuck there, but basically that waits on all the build jobs completing so if a disconnect happens things can be bad17:26
krotscheckodyssey4me: the script was moved over at 7:52AM PST17:26
jeblairmordred: (so that's an error from reprepro saying "i can't delete the package <k5start gobbledygook>"17:26
mordredmakes total sense17:26
clarkbgreghaynes: no disconnect happened from what I can see, but there was a failed build17:27
*** tdurakov has joined #openstack-infra17:27
odyssey4mekrotscheck until now we've been using pypi and a Rackspace wheel repo - I'm trying to change things up in the gate17:27
greghaynesclarkb: hrm, I thought I have failed handled fine there, let me check17:27
openstackgerritMerged openstack-infra/shade: Fix for stable/liberty job
odyssey4mekrotscheck in the hope of reducing build time17:27
krotscheckodyssey4me: Ur.... are you running all that on our trusty jenkins slaves? Because if so, the mirror update has been added to the global pip.conf17:28
krotscheckIn other words, you should already have it.17:28
clarkbFailed to connect to port 80: Connection refused is what caused the build to fail17:28
fungiodyssey4me: worth noting, the current wheel mirror is only building the transitive dependency set defined by our openstack global requirements list17:28
krotscheckThat too17:28
*** jsavak has quit IRC17:28
odyssey4mekrotscheck fungi Ah ok. and how complete is the pypi mirror?17:29
clarkbgreghaynes: I will look into the build failure if you can look into the state machine around wait for built images17:29
mordredodyssey4me: the pypi mirror is complete17:29
pleia2so much email piled up, almost forgot to say hello, good morning!17:29
nibalizeryou too pleia217:29
mordredodyssey4me: you should always use the pypi mirror that's configured for your nodes - it's local and complete and consistent and whatnot17:29
greghaynesclarkb: yea, fwiw improves that state machine a lot17:29
greghaynesso my brain is conflating that a bit17:30
*** mingyu has quit IRC17:30
clarkbgreghaynes: ya I think we should just go ahead and continue to get that in today17:30
mordredodyssey4me: you should also in that config also get the wheel mirror, which will have wheels prebuilt for all of openstack/reuqirements17:30
clarkbgreghaynes: since it should help debugging and straggler issues17:30
*** davideagnello has joined #openstack-infra17:30
odyssey4mekrotscheck it came down to wondering why it was that was not selected above on a pip wheel repo build which the wheel repo being in the extra index17:30
*** mingyu has joined #openstack-infra17:30
krotscheckodyssey4me: because the directory name wasn't normalized.17:31
clarkbkrotscheck: we have to rewrite -'s to _'s right?17:31
greghaynesclarkb: So it looks like workFailed is handled but workException is not17:31
clarkbodyssey4me: also fwiw we are using PyMySQL pretty effectively now17:31
fungiyeah, and lowercase all uppercase characters17:31
greghaynesclarkb: I want to say that we should have sent a workFailed in that scenario, though17:31
krotscheckclarkb: [-_.]+/-17:31
krotscheckclarkb: / [-_.]+/-/17:31
*** dims has joined #openstack-infra17:32
funginormalization includes lower-case transformation, right?17:32
BobBallclarkb: Sorry - which build is this?17:32
*** esikachev has joined #openstack-infra17:32
krotscheckIt's all in there.17:32
clarkbBobBall: our image builds17:32
greghaynesclarkb: yep, we should have sent workFailed17:32
greghaynesclarkb: so its possible that is a red herring17:32
jeblairclarkb, fungi: this is the normalization patch:
BobBall*confused* What are you pulling from ?17:32
odyssey4memordred I'd love to use the image configured /etc/pip.conf, but we're building many containers and dropping configs there... so my initial adventure is to read the configs and to use them with our current build paradigm. If that doesn't work then I'll change out pip config paradigm to be able to slurp in the whole pip.conf from the host17:33
greghaynesclarkb: another thing we should do regardless IMO is put a timeout on these jobs completing17:33
clarkbBobBall: the cirros image for your jobs I think17:33
fungijeblair: thanks, looks complete to me17:33
clarkbBobBall: they are listed in devstack so get cached17:33
odyssey4meclarkb yep, as are we for openstack services - but fun fact, Ansible requires MySQL-python, as does the MariaDB client17:33
jeblairodyssey4me: we really need you to use the mirrors we have configured17:33
clarkbgreghaynes: 2016-02-07 15:06:57,124 DEBUG nodepool.NodePool: DIB Image image-build (id 176259) failed to build. Deleting.17:33
BobBallclarkb: Ah - right.  Makes sense.  I assumed they were only pulled down if the driver was xenapi17:33
clarkbBobBall: we don't do per driver image builds17:33
jeblairodyssey4me: we can't really manage a system of this complexity with individual projects doing their own things17:34
clarkbgreghaynes: so the nodepool daemon did reqcord it as failing to build17:34
odyssey4meclarkb to clarify, Ansible requires that lib only when you use the mysql module17:34
clarkbgreghaynes: I think the wait for built image may just not realize it is done waiting when that happens17:34
jeblairodyssey4me: so if you need to run pip in a container, please use the pip.conf on the host17:34
BobBallclarkb: but none of infra's images actually use xenapi so it doesn't need to be cached at all :)17:34
BobBallclarkb: that's where I got confused17:34
clarkbBobBall: then please remove it from devstack17:34
clarkbBobBall: we cache what devstack tells us to cache17:34
BobBallclarkb: the XenServer third party CI tests it within devstack17:34
mordredyah. because we can't know at cache build time what runtime jobs are going to want to select17:35
odyssey4mejeblair Yep, I'm working on converging there as quickly as possible.17:35
jeblairodyssey4me: cool, thanks17:35
greghaynesclarkb: hrm, that is handled by almost the same code path - when the WatchableJob gets onWorkFailed it sets it's event which cleans it out of the queue being waited on17:35
BobBallso it is used, just not by -infra jobs17:35
*** vgridnev has joined #openstack-infra17:35
greghaynesclarkb: We can add some debug logging too17:35
fungiBobBall: clarkb: devstack plugins can define additional files to retrieve right? and we'll properly skip those since they're not in the vanilla devstack file list output17:35
mordredBobBall: is there a devstack plugin for xenserver?17:35
mordredyah. what fungi was getting at17:35
BobBallmordred: No; it's part of (predated plugins).  I presume that converting to a plugin would help avoid you caching it17:36
mordredBobBall: yah. honestly even if there as just a plugin that the only point of it was downloading this image17:37
mordredBobBall: (I know full devstack plugin migation would be likely non-trivial)17:37
*** ccrouch has joined #openstack-infra17:37
clarkbgreghaynes: I don't see where _image_build_jobs has the job remvoed on failure17:37
clarkbgreghaynes: we remove it from the db17:37
greghaynesclarkb: JobTracker17:38
*** claudiub has quit IRC17:38
greghaynesclarkb: that class auto sets up the cleanup whenever a job is added to it17:38
BobBallmordred: I have a feeling that the vmware disk is also cached and that's a lot bigger - now that we're talking about it I think I've seen that before17:38
jeblairfungi, AJaeger: looks like jenkins-jobs is exceeding its timeout.  i will merge all of the devstack timeout related changes then run in screen.17:39
clarkbgreghaynes: I think I see it. failure doens't imply completion in watchable job17:39
mordredBobBall: yes, I believe you are correct17:39
fungijeblair: thanks, and i'm not at all surprised17:39
clarkbgreghaynes: so that'll do it17:39
krotscheckjeblair, clarkb: Alright, i'll need a +A on and someone to go shut down the old mirrors.17:39
fungijeblair: did i review that one?17:39
greghaynesclarkb: it does? it calls self._event.set17:39
* krotscheck can try to shut them down, but without access i'd have to go the blackhat route.17:40
jeblairfungi: i think it was yolanda and i; topic:timeout17:40
clarkbgreghaynes: thta just means the job is done running on the wait17:40
BobBallmordred: Actually - as a proposal - how about we just dynamically set the IMAGE_URL in the existing nova plugin?  Does your caching script just scrape for the URLs from stackrc?17:40
clarkbgreghaynes: but it doesn't call the completion handler17:40
jeblairfungi: they were in stages, so i was hoping if i merged 1 at a time it would be ok17:40
fungikrotscheck: sorry, i reviewed it and forgot to vote. approved now17:40
krotscheckfungi: Sweeeeet17:40
clarkbgreghaynes: it iterates over distinct lists on failure and completion for the handlers17:40
greghaynesclarkb: ah, yes. So, that should still not cause this, but that is also a bug17:41
clarkbgreghaynes: and JobTracker uses completion to remove from the list17:41
fungikrotscheck: i'll delete the servers themselves later today17:41
greghaynesclarkb: since once you set the event it should return immediately, so we wont block on it17:41
clarkbgreghaynes: it would cause this if it caused waitForBuiltImages to never return17:41
krotscheckfungi: \o/17:41
clarkbgreghaynes: which it will17:41
krotscheckgreghaynes: We have finished wheels!17:41
clarkbgreghaynes: since the list that wait waits on never goes to zero17:41
fungikrotscheck: zoom zoom!17:42
clarkbgreghaynes: while len(self._image_build_jobs.running_jobs) > 0:17:42
greghayneskrotscheck: nice!17:42
greghaynesclarkb: oh! so its busy waiting17:42
greghaynesclarkb: is it spinning in a loop?17:42
*** cgalan has joined #openstack-infra17:42
fungikrotscheck: what shape are the npm mirror patches in?17:42
greghaynesclarkb: I bet that is also the cause of our gearman disconnects if soe17:42
rm_workI'm trying to get requirements back into shape for Octavia so we can get back in the checked projects list... we've got a patch to remove ndg-httpsclient up, but can we get this merged in Requirements since we really shouldn't roll back the other reqs change?
clarkbgreghaynes: yes thats what my paste showed, that is our cron thread from a day ago17:42
*** flepied has quit IRC17:43
krotscheckfungi: Not AFS-able17:43
greghaynesclarkb: yea, I mean 100% cpu17:43
clarkbgreghaynes: it has a half second sleep so shouldn't pin cpu17:43
*** cgalan has quit IRC17:43
krotscheckfungi: So, nooooo touchey17:43
*** yamamoto has joined #openstack-infra17:43
jeblairkrotscheck: like, yet, or ever?17:43
krotscheckAlso, they've bitrotted.17:43
*** baoli has joined #openstack-infra17:43
*** baoli has quit IRC17:43
krotscheckjeblair: yet.17:43
fungikrotscheck: right, i figured they'd bitrotted and been left behind by the afs switch, just didn't know if you'd started updating any yet17:43
*** boris-42 has quit IRC17:43
greghaynesclarkb: oh, I was conflating in my brain how it is now in
*** pvaneck has joined #openstack-infra17:44
krotscheckjeblair: The mirror project provides hooks which I can use to have it automatically write to an AFS directory, I just need to write the plugin.17:44
greghaynesclarkb: (notice the change of while to a for)17:44
zxiiroelectrofelix: no worries. i made a mailing list post since I think more folks should be aware of it.17:45
jeblairkrotscheck: oh that sounds nice17:45
clarkbgreghaynes: ah yup it should be fixed in that change17:45
*** kzaitsev_mb has joined #openstack-infra17:45
clarkbgreghaynes: I will just get that merged and restart service then17:45
greghaynesclarkb: ++17:46
openstackgerritMerged openstack-infra/system-config: Use k5start even for the readonly reprepro actions
clarkbgreghaynes: how important is it to get the child of that change in too?17:46
*** baoli has joined #openstack-infra17:46
*** e0ne has quit IRC17:46
clarkbgreghaynes: should we worry about those fialing tests or just do the parent for now?17:46
*** dahoo has joined #openstack-infra17:46
*** baoli_ has joined #openstack-infra17:46
openstackgerritMerged openstack-infra/project-config: Add missing semicolon in
greghaynesclarkb: the child only handles a case that comes up during gear disconnect17:47
greghaynesclarkb: so it isnt super important17:47
*** yaume has quit IRC17:47
openstackgerritMerged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 3]
greghaynesit handles the case where we are disconnected and submit a job before the client reconnects17:48
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-nodepool: Fix periodic log cleanup
clarkbjeblair: fixes the current issue with nodepool image builds. ps8 addressed your comments nad ps9 was a rebase to handle conflicts from bumping the gearman timeouts17:48
jeblairclarkb: thanks, i started reviewing it about 2 minutes ago :)17:49
clarkbBobBall: devstack provides a script that you can run to get all of the images to cache17:49
clarkbBobBall: so any changes should likely go there17:49
AJaegerthanks, jeblair !17:49
krotscheckjeblair: But, frankly, I've been so focused on mirror things that _everyting_ else on my plate has rotted17:50
*** Guest55447 has quit IRC17:51
*** baoli has quit IRC17:51
fungikrotscheck: well, you've done an awesome job on mirror things17:51
krotscheckfungi: Thanks :)17:51
anteayahear hear17:51
*** kzaitsev_mb has quit IRC17:51
jeblairkrotscheck: fermented, not rotted.  :)17:51
fungikrotscheck: thanks right back--this is already having a huge positive impact on stability and performance17:51
openstackgerritMerged openstack-infra/system-config: Removed pypi.<region> mirror manifests
krotscheckfungi: I can see the performance graphs, is there a similar graph that maps gate stability?17:53
*** sridhar_ram has joined #openstack-infra17:53
fungikrotscheck: just the elastic-recheck trending for known bugs about connecting to pypi probably17:53
openstackgerritMerged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 4]
*** cgalan has joined #openstack-infra17:54
krotscheckfungi: Coooooolio. Checking17:54
fungikrotscheck: though i guess that was partly solved when we implemented our previous pypi mirrors17:54
fungikrotscheck: so if there are stability improvements directly from the wheel mirrors, it'll likely be around jobs that used to time out trying to build wheels at install time17:55
openstackgerritMerged openstack-infra/project-config: Increase devstack timeout
krotscheckfungi: Yeah, I'm not seeing much in grafana17:55
krotscheckI mean logstash17:55
krotscheckEither way, 20 minute slice off of devstack is aesome enough.17:55
* krotscheck wonders if anyone outside of infra will notice.17:56
*** armax has joined #openstack-infra17:56
*** shashank_hegde has joined #openstack-infra17:56
fungikrotscheck: the big stability win will be now that we've got an extensible mirror structure and are adding distro package mirrors, we should soon see a marked drop in job failures due to inconsistent or unreachable provider/upstream mirrors17:56
*** fawadkhaliq has joined #openstack-infra17:56
*** asselin__ has left #openstack-infra17:56
*** tzn has joined #openstack-infra17:57
clarkbhowever we shouldn't expect much of a speedbump from those because we should already cache the majority of the packages on the images themselves17:57
krotscheckfungi: I can't claim credit for that one, jeblair did most of the work.17:57
clarkbwheels give us performance, apt/yum/etc mirrors give us stability17:57
*** daemontool has quit IRC17:58
*** daemontool has joined #openstack-infra17:58
clarkbrm_work: the requirements project has a set of core reviewers, infra tends to be hands off on that repo unless it is gate breaking17:59
*** sigmavirus24 is now known as sigmavirus24_awa17:59
rm_workAh, ok17:59
rm_workI thought those people were here17:59
rm_workDo they have their own channel?17:59
clarkbI think many of them do hang out here and in qa and in stable17:59
AJaegerjeblair: could you toggle your +A on ,please? The dependent job merged...17:59
jeblairrm_work, clarkb: #openstack-dev would be a good place to reach them17:59
clarkbI do not know of a dedicated channel, just want to point out that the infra cores probably won't approve that change18:00
*** trown is now known as trown|lunch18:00
jeblairAJaeger: ack18:00
rm_workclarkb: ah yeah no worries18:00
*** annegentle has joined #openstack-infra18:00
rm_workI just remember asking in here last time for global-reqs and it being effective18:00
AJaegerregarding stability and performance: I noticed that we have a long *post* queue and short *gate* according to zuul status. That's quite a difference. And also 78 jobs in the post queue - older 4hours - sounds a lot jobs merged18:01
AJaegerkrotscheck, mordred, jeblair: Great job, thanks!18:01
*** kzaitsev_mb has joined #openstack-infra18:02
*** rossella_ has quit IRC18:02
AJaegerregarding timouts: Should I abandon my ironic change to decrease the timeouts now - or rebase it?
*** rossella_ has joined #openstack-infra18:03
*** vivekd_ has joined #openstack-infra18:03
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
mordredjeblair: how does that look? ^18:03
*** vivekd has quit IRC18:05
pleia2ooh debian, the things I miss with my globetrotting18:05
*** vivekd_ is now known as vivekd18:05
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Remove unecessary session.commit()
AJaegerjeblair, mtreinish: how shall we continue with the pipeline merges (remove -qa/-neutron)? and ?18:06
AJaegerwelcome back, pleia2 !18:06
jeblairclarkb, greghaynes: lgtm; i noted ^ but just fixed it in a followup rather than send it around again.18:06
anteayayes hello pleia218:06
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
*** armax has quit IRC18:07
clarkbjeblair: kk will look at that in a few18:07
*** derekh is now known as derekh_afk18:07
AJaegerpleia2: regarding globetrotting: The first major part of the translations work is in and documented in . Could you put this on your review list for the week, please?18:08
*** imcsk8 is now known as imcsk8|afk18:08
*** krtaylor has quit IRC18:08
*** doug-fish has quit IRC18:08
pleia2AJaeger: woo! and added to the list, thanks :)18:08
*** doug-fish has joined #openstack-infra18:09
*** daemontool has quit IRC18:09
jeblairmordred: i think i see 1 thing in 27751618:09
*** daemontool_ has joined #openstack-infra18:09
AJaegerpleia2: amotoki deserves most of the credit for the implementation18:09
openstackgerritMerged openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support
pleia2AJaeger: they were the one making a lot of comments in your reviews, yeah?18:11
*** armax has joined #openstack-infra18:12
*** jsavak has joined #openstack-infra18:12
fungijeblair: do you have a sec to help me dig into mailman logs? the release announcement job actually delivers the message to lists.o.o and i can in turn see exim there successfully handing off to the mailman router, but find no corresponding log entries from mailman itself: 2016-02-06 16:03:44 1aS5Kq-0007AT-MO => openstack-dev <> R=mailman_router T=mailman_transport18:12
*** Sukhdev has quit IRC18:12
jeblairfungi: yes18:12
*** doug-fish has quit IRC18:13
fungijeblair: the corresponding job log is
fungibut after mailman gets the message it seems to me to go into a black hole18:13
jeblairfungi: i don't see a message-id in the exim logs, which is okay -- exim will add one based on the queue id, so we should be able to search on 1aS5Kq-0007AT-MO in the mailman logs18:14
*** shardy has quit IRC18:15
AJaegerpleia2: amotoki reviewed heavily as well, yes.18:16
fungijeblair: i tried that, to no avail18:16
*** jpr has joined #openstack-infra18:16
fungijeblair: i also grepped through them by sender address18:16
*** jsavak has quit IRC18:16
greghaynesjeblair: awesome, ty18:16
*** ociuhandu has quit IRC18:16
*** ociuhandu has joined #openstack-infra18:17
*** [HeOS] is now known as HeOS18:18
harlowjamordred yt jeblair and i were talking about said supposed talk-to-yahoo-about-openstack-infra sometime in the future last week, but u were traveling or something18:18
harlowjaperhaps can somehow through the talk (which is still ongoing internally) about y! donating resources to the infra-team in that same meeting/talk/blah blah18:19
*** krtaylor has joined #openstack-infra18:20
harlowjaso ya we should do it (said talk) :-P18:21
*** annegentle has quit IRC18:22
mgagneI found oauth token to work fine for creating repositories on GitHub contrary to what this comment states:
*** bpokorny has joined #openstack-infra18:22
mgagnehow can I update it so I don't break the world?18:23
mgagnemordred ^ since your name shows in git blame18:23
jeblairfungi: i think we have an extremely high number of files in /var/lib/mailman/qfiles/bounces18:23
*** markmcclain has quit IRC18:24
fungioh, ew. i had looked in the bounces logs, but not the qfiles18:25
*** markmcclain has joined #openstack-infra18:25
jeblairfungi: don't know if that's related yet...18:25
mordredmgagne: neat - I think it maybe did not used to work18:25
jeblairdon't really know anything without ls returning18:25
mgagnemordred works with github enterprise 2.1.3 with following scopes: repo, user, write:org18:25
*** yamamoto has quit IRC18:26
fungijeblair: i have interrupted my ls|wc -l in that case ;)18:26
jeblairfungi: i'll let you know what mine reports :)18:26
fungijeblair: thanks!18:26
*** annegentle has joined #openstack-infra18:26
*** lucasagomes is now known as lucas-dinner18:28
*** dtantsur is now known as dtantsur|afk18:28
*** nadya has joined #openstack-infra18:28
*** nadya is now known as Guest4490118:28
*** matrohon has joined #openstack-infra18:29
*** eil397 has joined #openstack-infra18:29
openstackgerritMonty Taylor proposed openstack-infra/system-config: Remove restricted and multiverse
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Remove tooz testing of etcd/consul on liberty or kilo
fungimordred: rebase?18:31
jeblairmordred: oops, i closed your screen session18:31
fungiof 27717318:31
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Remove tooz testing of etcd/consul on liberty or kilo
jeblairmordred: sorry18:31
mordredfungi: yes. it had merge conflict18:31
*** armax has quit IRC18:31
*** flepied has joined #openstack-infra18:31
*** exploreshaifali has joined #openstack-infra18:32
jeblairfungi: 86856018:32
mordredmgagne: any idea if it works with public github?18:32
*** Guest44901 has quit IRC18:32
jeblairfungi: list in ~root/bounces18:32
fungijeblair: that's a very large number, yes18:32
*** jsavak has joined #openstack-infra18:33
fungijeblair: those are message-ids?18:33
mgagnemordred I can test. after further test, it only needs "repo or public_repo" and write:org scopes.18:33
mgagnemordred testing with github pub now18:33
mordredmgagne: I think the best way to fix this would be to add an option to the config file for adding an oauth $something18:33
mordredmgagne: (not sure what the oauth python workflow for this would look like)18:34
*** baoli_ has quit IRC18:34
mordredmgagne: and update manage-projects to use oauth creds if they're there, and username/password otherwise18:34
mgagnemordred there is a way to make it work in a backward compat way but would need lot of logic18:34
*** baoli has joined #openstack-infra18:34
*** savihou has joined #openstack-infra18:34
mgagnemordred like if your oauth token doesn't have right scopes...18:34
*** kdas_ has joined #openstack-infra18:35
jeblairfungi: i don't think so -- i think mailman names its pickle files through some kind of timestamp/uid thing... i don't know whether/how to associate those with anything else18:35
mgagnemordred I found jeepyb not to be that much verbose when it comes to error handling18:35
*** kushal has quit IRC18:35
mordredmgagne: yah - it's not one of my finest pieces of code18:35
fungijeblair: the prefix looks like unix time, so i'm looking for things around 145477464018:35
jeblairfungi: i do not see a bounce runner running18:35
*** eil397 has quit IRC18:35
mordredmgagne: the logic difference really should just be in the constructor, right?18:36
*** eil397 has joined #openstack-infra18:36
mordredmgagne: so you should be able to do "if oauth in config: github(oauth) else: github(username,password)" right?18:36
mgagnemordred yes18:36
*** baoli has quit IRC18:36
mordredoh - that looks great18:37
*** matrohon has quit IRC18:37
mordredI see no problem landing that as it is upstream18:37
jeblairfungi: i think we should remove (or move) those files and declare bankruptcy on bounces and restart mm18:37
mgagnemordred will do now18:37
fungijeblair: i absolutely agree there18:37
*** slogan621 has joined #openstack-infra18:38
jeblairfungi: i await the results of your search for any clues regarding our lost message18:38
fungijeblair: there are a few files in the right time range to be candidates so i'll let you know in a sec18:38
*** arxcruz has quit IRC18:38
*** matrohon has joined #openstack-infra18:39
openstackgerritMathieu Gagné proposed openstack-infra/jeepyb: Use oauth token to create GitHub project if available
fungijeblair: /var/lib/mailman/qfiles/bounces/1454774626.386724+22e4b301acc4b97bf2c1d61245aaa5650134fb89.pck18:41
*** trown|lunch is now known as trown18:42
*** tzn has quit IRC18:42
*** abregman is now known as abregman|afk18:43
fungijeblair: "You have to be a subscriber to post to this mailing list, so your\nmessage has been automatically rejected."18:44
fungijeblair: thanks, now i know where to dig for this stuff18:44
*** e0ne has joined #openstack-infra18:44
jeblairthat'll do it18:44
*** armax has joined #openstack-infra18:44
jeblairfungi: and it got stuck there since the bounce runner isn't running18:44
* AJaeger just blogged about templates in Zuul - . If there's anything that should be added to infra manual, feel free to propose a change or tell me18:45
fungijeblair: right, which is why i didn't find it in the bounces log18:46
fungijeblair: okay, so i should move /var/lib/mailman/qfiles/bounces elsewhere and create an empty directory there, then restart the mailman initscript?18:47
*** piet has quit IRC18:47
*** _amrith_ is now known as amrith18:48
jeblairfungi: oh, i think it may be slightly more complex -- i think announce -> list as queue id 1aS5Kq-0007AT-MO; list rejects due to sender not subscribed; rejection goes to exim as queue id R=1aS5Ks-0007GV-3e; exim can not deliver it because no-reply@o.o does not exist; exim sends bounce to mailman as queue id 1aS5Ks-0007Gy-8U; then *that* gets stuck in the bounce queue because there is no runner18:48
odyssey4menice work there AJaeger :)18:50
jeblairfungi, mordred: maybe we can come up with an ansible play to run jjb on all the jenkins masters?18:52
AJaegerthanks, odyssey4me18:52
*** jsavak has quit IRC18:52
fungijeblair: that sounds like a thrilling convenience18:52
odyssey4meAJaeger now you can just tell people to rtfm :p18:52
fungi(compared to starting screen sessions on 8 servers and having to track them all)18:52
*** jsavak has joined #openstack-infra18:53
mordredjeblair: yes. I think that sounds completely sane18:53
jeblairi'll see if i can whip something up18:53
fungijeblair: any objection to my plan for moving/recreatign the bounces dir?18:54
jeblairfungi: oh missed that, sorry.  lgtm.18:54
*** esikachev has quit IRC18:54
*** esikachev has joined #openstack-infra18:54
fungilooks like it's mode 2770 owned list:list18:56
AJaegerodyssey4me: we hae the infra manual for rtfm but too often people still ask - that's why I'm aksing here for feedback what's missing there and should be added...18:58
fungijeblair: /var/lib/mailman/qfiles/bounces.old is the old one18:58
fungican recursively delete once we're comfortable18:58
AJaegerodyssey4me: but yes, sometimes a different way to present content might help as well...18:58
odyssey4meAJaeger hahaha, I made a comment the other day to someone that people don't ready manuals, but they do read blog posts!18:58
*** salv-orl_ has joined #openstack-infra18:58
nibalizerjeblair: ++18:59
odyssey4meI do think that perhaps it has something to do with the search ranking of the manuals though. Blogs often rank higher.18:59
*** jsavak has quit IRC18:59
jeblairnibalizer, fungi, mordred:   is that all we should do?18:59
*** jsavak has joined #openstack-infra19:00
*** matrohon has quit IRC19:00
mordredjeblair: yes, that looks good19:00
pleia2odyssey4me: I think you're right19:00
nibalizermaybe update project-config in that playbook?19:00
nibalizeror just use the version puppet laid down I guess?19:00
pleia2odyssey4me: maybe we should write blog posts that point to docs :D19:01
jeblairnibalizer: yeah, i'd rather use the latter -- this is mostly for when we know that jjb has timed out on its normal run19:01
mordredjeblair: might be worth looking at
jeblairmordred: oh, is ansible going to have a default timeout or anything?19:01
jeblairmordred: heh19:01
*** salv-orlando has quit IRC19:01
*** amrith is now known as _amrith_19:02
mgagnemordred works fine with public github for me19:02
nibalizerjeblair: oh so you don't see this running out of cron on the bastion and taking over for that puppet exec that runs it now19:02
nibalizerthat wfm19:02
mordredjeblair: so async: 3600 poll: 30 or something - then you can run ansible-playbook -f 8 your-playbook.yaml and it _should_ work19:02
*** sigmavirus24_awa is now known as sigmavirus2419:02
jeblairnibalizer: right, this is corrective action for exceptional circumstances19:02
fungijeblair: the --conf is unnecessary but i suppose being explicit is fine19:02
nibalizermordred: sync is probbaly ok ?19:02
AJaegerpleia2: should I add more links next time? ;)19:02
nibalizermaybe sync for the first one then async for the rest19:03
mordrednibalizer: the issue is the timeout19:03
nibalizerif something is wrong we don't want to break them all at the same time19:03
fungijeblair: it would also help to git pull --ff-only or git reset --hard origin/master in /etc/project-config first19:03
nibalizersame rough reasoning is in clarks restart script19:03
mordredright - but this is re-running jjb19:03
mordredwhichis normally done just by opening 8 screen sessions19:03
jeblairfungi: that's what nibalizer was suggesting, but i was thinking it should already be updated usually...19:03
pleia2AJaeger: haha, maybe!19:04
mordredand letting them all go and periodically eyeball checking19:04
jeblairfungi: is 1 hour long enough?19:04
fungijeblair: aside from that, as long as it's running as root and with /root as $HOME it should be equivalent with what i've been doing19:04
jeblairfungi, mordred, nibalizer:
fungijeblair: it really depends on how excessive the job config update is. i've had jenkins01 and 02 take upwards of 90 minutes to complete (or to finally bomb out on a jenkins api error, more freuquently)19:05
jeblair7200 it is19:05
*** tdurakov has quit IRC19:05
nibalizerjeblair: lgtm19:05
fungiyeah, that should be fine. the only times it's necessary to make sure project-config updates are in place is if puppet is being separately laggy, or in dire/urgent situations where you don't want to wait for puppet to get around to firing it once19:06
jeblairokay, it's off and running; proposing patch now19:07
fungior perhaps if you've disabled puppet on the servers, but in those cases you may also want to separately override project-config to a specific commit anyway19:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Lint YAML files
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix linting errors in YAML files
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add playbook to run JJB on all jenkins masters
AJaegerjeblair, mordred: I rebased those yamllint changes. jeblair I looked yesterday closer at them, I think the stylistic issues are all disabled now and we should give this is a try ^19:10
AJaegermordred: you still have a -2 on 27184419:10
jeblairAJaeger: why?19:10
nibalizercrinkle: mordred after lunch would you like to work on moving infracloud-west into management?19:10
nibalizerI believe we are nearly there19:10
nibalizerclose enough to JDI19:10
AJaegerjeblair: it will fix issues like duplicate keys19:10
jeblairAJaeger: when has that happened?19:11
AJaegerjeblair: seldom ;)19:11
jeblairAJaeger: i'm not convinced it's useful.19:12
AJaegerjeblair: I'm fine with asking to abandon as well - or merge and revert if it causes problem, I just think we should flip the coin and not let him rebase too often19:12
jeblairAJaeger: most of our yaml is structured to avoid the temptation to have duplicate keys.19:12
AJaegerjeblair: indeed19:12
fungialso i still feel like duplicate key checking should be the realm of the yaml parser itself, if the pyyaml devs ever get around to making an option for that19:13
crinklenibalizer: I added an agenda item for tomorrow's meeting, maybe we should wait for more discussion to happen?19:13
jeblairfungi: indeed.19:13
fungii occasionally check back on that bug, and then get depressed that it's still going nowhere19:14
AJaegerfungi: I agree, that would be the best place.19:14
fungias in there still seems to be no consensus from pyyaml upstream that it's desirable to check for duplicate keys19:14
AJaegerI'll add a comment asking to abandon, saying that the cases it covers are rarely hit...19:14
nibalizercrinkle: i mostly just mean putting the infracloud controller and infracloud compute hosts under bastion control19:14
fungiand now there's a fork of pyyaml which does error on duplicate keys, but it also comes with tons of other baggage dependencies19:14
nibalizerso having ansible run puppet on them19:14
nibalizernot turning on nodepool19:15
crinklenibalizer: okay19:15
nibalizerthat work is going to be a bit touchy19:15
*** doug-fish has joined #openstack-infra19:16
*** sbalukoff has joined #openstack-infra19:16
*** jgriffith is now known as jgriffith_away19:16
jeblairmordred: legit failure at
openstackgerritSpencer Krum proposed openstack-infra/publications: ConfigMgmtConf 2016 by Spencer Krum (nibalizer)
mordredjeblair: GAH19:17
jeblairmordred: left comment on what i think the fix is19:17
nibalizerhow do i git review to a target branch?19:18
jeblairnibalizer: git review <branch>19:18
nibalizeroh just branch19:18
mordredjeblair: you are correct19:18
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
jeblair(and git restack follows the same pattern)19:18
fungiis this what the openstack health dashboard is supposed to look like?
*** rbrndt_ has quit IRC19:18
nibalizerfungi: i think thats the api server19:19
AJaegerfungi: the URL looks correct, the JSON not19:19
*** adrienverge has joined #openstack-infra19:19
clarkb is the dashboard19:19
clarkbhealth.o.o is just the rest api19:19
fungiaha, thanks clarkb19:19
pleia2nibalizer: referring to handling publication changes?19:19
mordredjeblair: oh - you know - we should maybe make the lockfile in the cron different too19:19
nibalizerpleia2: hahaa yep19:20
pleia2I wrote this handy thing :)19:20
mordredjeblair: I'll need to spin up another flock screen for it19:20
nibalizeri think you link this to me every time i start flailing19:20
pleia2nibalizer: I mess it up every time, so docs++19:21
adrienvergeAJaeger: About #271845, could you explain the reason? Sorry I wasn't on irc19:21
*** doug-fish has quit IRC19:21
*** pradk has quit IRC19:21
AJaegeradrienverge: read backscroll at
adrienvergeOK, just read on eavesdrop.openstack.prg19:22
nibalizeralso minor announcement, my talk ^^ that talk, got noted in the DevOps Weekly newsletter19:22
nibalizerso thats pretty cool19:22
pleia2nibalizer: nice, congrats19:22
*** doug-fish has joined #openstack-infra19:22
pleia2nibalizer: is there video?19:22
AJaegercool, nibalizer !19:22
adrienvergethe key duplication actually happens (for instance, there are currently some in heat-templates, etc.)19:22
* pleia2 sees \o/19:23
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
mordredjeblair: ^^ ok - I'm now holding both locks19:23
AJaegeradrienverge: but not in project-config...19:24
adrienvergeAnd upstream (pyyaml) wont provide such a "key duplication" detection feature19:24
*** ifarkas has quit IRC19:24
jeblairmordred: if you head down that road, we need to make unref distro-specific19:24
adrienvergeAJaeger: That's true19:24
*** bardia has joined #openstack-infra19:25
adrienvergeAJaeger: but enabling this linter has little cost, and makes sure it won't happen like in other projects19:25
bardiaanyone which channel to go to for tempest issues?19:25
jeblairmordred: anything else that might be shared between them?  i'm thinking most of it has moved into afs now...19:25
jeblairbardia: #openstack-qa19:25
clarkbpyyaml actually says you should modify the yaml parser class to detect it if you need it19:25
clarkbsalt does this19:25
*** Sukhdev has joined #openstack-infra19:25
mordredjeblair: I cannot parse your sentence19:26
mordredjeblair: unref?19:26
bardiajeblair: thanks19:26
AJaegeradrienverge: as jeblair said: This does happen so rarely that we don't need to check for it19:26
jeblairmordred: see comment19:26
clarkbit is fairly simple code iirc, you keep a hash table of keys and check if any key already exists as you go19:26
jeblairclarkb: i would not be opposed to our tools growing that, or even us releasing a little lib to automate that.19:26
*** armax has quit IRC19:26
mordredjeblair: AH19:26
*** electrofelix has quit IRC19:26
adrienvergeAJaeger: alright, got it19:27
* mordred has idea19:27
adrienvergeclarkb: Sure, but we don't want to do it in oopenstack-infra when tools exists19:27
jeblairclarkb, adrienverge: because a program should be able to detect its own syntax errors19:27
fungiclarkb: i think you'd need to subclass part of pyyaml if you want to deep check that on load (nested associative arrays). i looked into it briefly at one point19:27
clarkbfungi: yup, that is what salt does19:27
clarkband jjb does it for other reasons19:28
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
clarkblet me find the salt code19:28
mordredjeblair: how about that?19:28
mordredjeblair: (and nice catch)19:28
jeblairmordred: wfm19:28
mordredjeblair: it will be an ugly file name - but in the realm of things I care about - that's not one of them19:29
jeblairchanged: []19:29
jeblair1/8 complete :)19:29
clarkbjeblair: greghaynes you ok with me restarting nodepool now to pick up that latest disconnect handling patchset?19:31
greghaynesclarkb: go for it19:31
*** bpokorny_ has joined #openstack-infra19:31
jeblairclarkb: ++19:31
*** annegentle has quit IRC19:32
*** akshai has joined #openstack-infra19:32
*** bpokorny_ has quit IRC19:34
*** bpokorny has quit IRC19:34
clarkbbluebox may have leaked floating IPs19:34
*** bardia has left #openstack-infra19:34
*** armax has joined #openstack-infra19:34
*** bpokorny has joined #openstack-infra19:34
clarkbmordred: ^ maybe we should make a test case for this with devstack/tempest then see if we can get it fixed19:35
clarkbhrm I wonder if restarting nodepool causes the leak. Oustanding requests may not be associated iwth an instance so when the instance gets deleted the fip remains?19:36
*** f1ller- is now known as filler19:36
*** armax has quit IRC19:36
*** esker has joined #openstack-infra19:37
*** esker has quit IRC19:37
mordredclarkb: yah. that's likely19:37
*** esker has joined #openstack-infra19:38
mordredclarkb: once image workers is stable and I can re-focus on shade-nodepool, the change in fip allocation model in shade should allow us to stop leaking fips that way19:38
clarkbalso I think we are really really close to that stable image builders19:38
mordredclarkb: since the fip create call is done with a server port in the first place, so we'll always have the tracking info19:39
mordredclarkb: awesome19:39
clarkbI am working on doing a 5 minute fip list diff and will cleanup the unused fips19:39
mordredclarkb: I'm almost done with apt mirrors, so will have brainspace to do shade-nodepool next19:39
prometheanfiremordred: who else is core in glean?19:39
mordredprometheanfire: greghaynes19:39
greghaynesI think SpamapS is now too19:39
*** fawadkhaliq has quit IRC19:40
SpamapSI am19:41
*** dprince has quit IRC19:41
SpamapSI need to hit the review queue today actually.19:41
prometheanfirethere are a couple of DIB fixes too, but I imagine it's in people's queues19:41
*** piet has joined #openstack-infra19:42
*** cloudtrainme has quit IRC19:42
openstackgerritMerged openstack-infra/system-config: Remove restricted and multiverse
*** tdurakov has joined #openstack-infra19:43
*** fitoduarte has joined #openstack-infra19:43
yolandahi SpamapS, i have several changes in glean and dib that need review19:45
yolandai heard about you checking the review queue...19:45
*** ociuhandu has quit IRC19:46
openstackgerritBrant Knudson proposed openstack-infra/project-config: Support stable requirements
prometheanfireSpamapS: you failed to be seen sir19:48
prometheanfireor more correctly failed to not be seen19:48
* SpamapS [boom]<wwaahhhhhh>19:48
clarkbbluebox floating IPs cleaned up19:49
SpamapSand now for something completely different (lunch)19:49
prometheanfireya, 10m til meeting here19:49
ianwprometheanfire: re your question some 6 hours ago ... finalise.d does run in the build chroot ... see
*** kzaitsev_mb has quit IRC19:50
*** jsavak has quit IRC19:50
prometheanfireianw: ah, ok, I'll move it there then19:50
prometheanfiredo I need to include the finalize element then?19:50
prometheanfireif there is one19:51
prometheanfirehaven't looked19:51
clarkbgreghaynes: those test fails on the cild change are pretty conssitent19:51
AJaegerjeblair, fungi: What do you want to do with removal of periodic-qa and periodic-neutron queues, now? mtreinish commented on and this might need another reply. Some of these periodic jobs are currently broken, the whole patch series fix them - so I'd like to know how to continue and would appreciate your help here, please19:51
*** jsavak has joined #openstack-infra19:51
clarkbgreghaynes: do you have time to look closer at them or should I do that19:51
*** rbrndt_ has joined #openstack-infra19:51
greghaynesclarkb: I cant right now...19:51
*** Sukhdev has quit IRC19:52
jeblairAJaeger: i replied on 31819:52
greghaynesAlthough I would really prefer that over debugging this pptp issue19:52
prometheanfireianw: I specificly don't want finalize.d because it happens after the copy19:52
ianwclarkb: you talking about the image deletion ( ?  i was having a look yesterday19:52
*** salv-orl_ has quit IRC19:53
clarkbianw: yup19:54
AJaegerjeblair: yes, you did. So, ok to +A your changes now?19:54
clarkbianw: did you see that I got fedora-23 updated everywhere late my friday?19:55
prometheanfireianw: so, should stay in post-install.d19:55
greghaynesianw: I left a reply to your comment on there btw19:55
clarkbianw: the root / was grown to 15GB which seemed small but I wasn't sure if that was making use of the entire available disk19:55
greghaynesianw: the fail you mentioned is a fail, but thats a known one I think that happens during cleanup. The real fail there is that we timed out (which kicked off the cleanup)19:56
*** nadya has joined #openstack-infra19:56
*** daemontool_ has quit IRC19:56
ianwclarkb: hmm, ok, well growroot should be filling up the disk.   i'll kick something off and check it out19:56
*** nadya is now known as Guest6986519:56
*** jsavak has quit IRC19:57
*** Sukhdev has joined #openstack-infra19:57
ianwgreghaynes: hmm, ok, it did look rather suspicious :)19:57
*** sfinucan has quit IRC19:57
*** tongli has quit IRC19:57
*** akshai has quit IRC19:57
*** jsavak has joined #openstack-infra19:58
*** Guest69865 has quit IRC19:59
ianwprometheanfire: well, there's some precedent for doing things like cleanup in finalise.d -- see existing redhat-common & ironic-agent elements.  another option maybe package-installs should move earlier (98) so it's possible to put another thing after it.  get some input from greghaynes/SpamapS20:00
prometheanfireif they respond to me :(20:00
*** jtomasek_ has quit IRC20:00
greghaynesIf you leave comments in the reviews i'll reploy when I get to it20:01
*** woodster_ has joined #openstack-infra20:02
*** akshai has joined #openstack-infra20:02
*** bpokorny has quit IRC20:03
*** arxcruz has joined #openstack-infra20:04
*** bpokorny has joined #openstack-infra20:04
ianwclarkb: -- 2016-02-07 14:48 : is that a day behind?  failed to build as timed out downloading a cirros image from citrix20:04
*** annegentle has joined #openstack-infra20:04
greghaynesianw: nodepool was wedged and not doign builds yesterday20:05
*** dprince has joined #openstack-infra20:05
*** vgridnev has quit IRC20:05
ianwgreghaynes: ahh, so situation normal ;)20:05
greghaynesianw: I think as a result of that20:05
*** ociuhandu has joined #openstack-infra20:05
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud
clarkbianw: yes we don't hae them as of today, but they were built late friday20:05
*** eil397 has quit IRC20:06
AJaegerdims, dhellmann: Could you review for a requirments related change in project-config, please?20:07
ianwclarkb: could you hold/put my key on fedora-23-rax-iad-7793913 so i can poke?20:07
clarkbianw: sure20:07
mordredjeblair: I believe once lands - I should remove the crufty files in /etc/reprepro, then we should unmount the volume in afs, rename it, make new mount point and mount it, yeah?20:08
dimsAJaeger : looking20:08
clarkbianw: can you point me to your public key?20:08
jeblairmordred: yep20:08
*** cgalan has quit IRC20:08
ianwclarkb: one ending in UcMr is really only one i need20:09
clarkbianw: I added all of them :) jenkins@
mroddenianw you working on bashate release?20:09
mordredjeblair: cool.20:09
ianwmrodden: i have a whole heap of stuff queued up for ages unreviewed20:10
mordredclarkb: you wanna +A 277516 for me? :)20:10
AJaegerthanks, dims20:10
mroddenianw, for bashate...?20:10
mroddenwe probably need to get out a release just for the pbr > 1.0 fix20:10
*** daemontool has joined #openstack-infra20:10
ianwmrodden: yes.  it fixes a whole lot of stuff
ianwmrodden: but i guess it needs a rebase at this point, i'll try to get to that asap20:11
clarkbmordred: its not urgent right? we don't debian yet20:11
mroddenwell we have plenty of delta already between 0.3.2 and the next release including a drop of py26 support20:11
clarkboh! you want the paths to be correct before using the mirror20:11
clarkbI see20:12
mroddenianw, we should just do it and then 0.4.1 when we get that fixed...20:12
*** cloudtrainme has joined #openstack-infra20:12
mordredclarkb: yah - it's less about debian and more about moving ubuntu20:12
clarkbmordred: should become a defined type otherwise you can only instantiate it once20:12
mordredbefore go-live20:12
clarkbmordred: you can do that in a follow up if you just want to ge tthe /ubuntu path in20:13
*** tdurakov has quit IRC20:13
*** jgriffith_away is now known as jgriffith20:13
mordredclarkb: let's do that as a followup - but good call20:13
*** armax has joined #openstack-infra20:13
ianwclarkb: this looks sad :( growroot[419]: find: missing argument to `-name'20:13
*** tdurakov has joined #openstack-infra20:13
*** rockyg has joined #openstack-infra20:13
*** tdurakov has quit IRC20:13
*** gildub has joined #openstack-infra20:14
*** esp has joined #openstack-infra20:14
clarkbso I will -1 based on ^20:15
openstackgerritBen Kero proposed openstack/diskimage-builder: Replace sfdisk partitioning with parted
mordredclarkb: wow. I totally thought I'd fixed that already20:16
*** tcammann has joined #openstack-infra20:16
openstackgerritMonty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
mordredclarkb: sorry about that20:17
*** ccrouch has quit IRC20:17
*** NobodyCa1 has joined #openstack-infra20:17
openstackgerritMerged openstack/diskimage-builder: Make dkms element depend on dkms package
*** armax has quit IRC20:17
*** armax has joined #openstack-infra20:18
*** adrienverge has quit IRC20:18
clarkbianw: once hte job on that host finishes you should be able to manually do a grow root to figure out what the invocation needs to be (maybe during job too)20:19
*** rockyg has quit IRC20:19
clarkbianw: then I guess it is patch dib and rebuild images?20:19
fungijeblair: mordred: so after i kinit and aklog as fungi i'm getting a permission denied trying to delete things under /afs/
fungijeblair: mordred: should i be using fungi.admin instead?20:19
ianwclarkb: yeah, seems it's not that easy ... the script works *outside* systemd.20:19
jeblairfungi: yes, your normal account doesn't have write perms20:20
jeblairfungi: only fungi.admin is in system:administrators20:20
ianwclarkb: maybe ordering ... udevadm might need something else up?20:20
fungiokay, thanks, thought that might be it but wanted to be sure before i possibly made a mess20:20
jeblairfungi: you can use "fs la /afs/" to check perms; and "pts membership system:administrators" to check membership20:20
mordredjeblair: one more tiny change in since your last +220:20
*** tcammann has quit IRC20:21
jeblairmordred: +320:21
*** esp has quit IRC20:21
mordredwhen it lands I will do the manual steps20:21
mordredjeblair: I may have to ask you some questions about the volume rename stuff20:21
mordredas I've never done that before20:21
fungijeblair: manpage suggests klog -pr fungi.admin should work but i'm getting an rpc error on that...20:21
jeblairfungi: aklog20:22
jeblairfungi: (no args needed)20:22
mordredjeblair: how does k5start work if there's no aklog involved?20:22
jeblairmordred: it gets tokens20:22
jeblairmordred: it's automagic20:22
*** NobodyCa1 has quit IRC20:23
fungijeblair: thanks! that seems to have probably worked... though now things are hanging when i try to rm a file. there are some nat workarounds needed too, yeah?20:23
ianwclrakb: yeah, i think i probably just got lucky in the ordering with my testing; i think i need to add some bits to the service file to make sure it starts after udevd sockets are up20:23
mordredfungi: you need the udp20:23
fungirealizing now why i probably didn't try setting this up on my workstation back when we first stood up afs20:24
clarkbianw: ok20:24
mordredfungi: but once you get it set up it's great20:24
ianwclarkb: thanks, done with fedora-23-rax-iad-779391320:24
fungimordred: thanks, i'll hunt around and see how restrictive i should be able to make the portmap for this20:24
*** akshai has quit IRC20:24
jeblairfungi: should be okay with a connection tracking firewall but you may need to increase the udp conntrack timeout20:25
fungioh, actually it eventually came back and prompted "rm: remove write-protected regular file ..." but when i said y it errored with "Permission denied"20:25
*** |-paul-| has joined #openstack-infra20:25
fungijeblair: default timeout for udp sessions on openbsd is crazy long but i'll check that20:26
*** dolpher has quit IRC20:26
jeblairfungi: output of klist?20:26
fungithis is on too so it's not the ro volume20:26
*** dolpher has joined #openstack-infra20:26
*** salv-orlando has joined #openstack-infra20:27
fungijeblair: oh! it's using fungi as my default principal20:27
fungido i need to kdestroy that one?20:27
jeblairfungi: yeah; maybe 'kdestroy'20:27
*** tiswanso has quit IRC20:27
jeblairthen 'kinit fungi/admin && aklog' should do it20:27
*** tiswanso has joined #openstack-infra20:28
fungiDefault principal: fungi/admin@OPENSTACK.ORG20:28
fungiand that will let me remove a file20:28
fungiokay, deleting the old non-normalized wheel trees now20:29
clarkbI have successfully built an image post service restart and have queued the uploads20:29
*** jlk has quit IRC20:29
clarkbjust making sure that things look good on demand and that image builds in general don't break again tomorrow morning20:29
*** jlk has joined #openstack-infra20:30
*** jlk has quit IRC20:30
*** jlk has joined #openstack-infra20:30
*** armax has quit IRC20:30
*** esikachev has quit IRC20:30
jeblairclarkb: i thought you made an imgur with "build an image post service"20:31
*** annegentle has quit IRC20:31
jeblairer built20:31
*** dolpher has quit IRC20:31
fungibetter than the old snapshot-based snapchat we used to have20:31
fungikrotscheck: jeblair: mordred: cleanup from is now done, if anyone wants to spot-check behind me20:31
*** annegentle has joined #openstack-infra20:32
*** cloudtrainme has quit IRC20:32
*** dolpher has joined #openstack-infra20:32
jeblairfungi: it deletes 2 messages ago.  and gets stuck for weeks at a time?20:32
clarkbjeblair: that happens after nodepool does images properly20:32
clarkbthat is actually the sort of application that makes perfect sense to host on swift20:33
clarkbthere is no real index20:33
clarkbupload, view, delete20:33
fungijeblair: heh, sounds about right20:35
*** ccrouch has joined #openstack-infra20:35
*** salv-orlando has quit IRC20:36
*** akshai has joined #openstack-infra20:36
*** vivekd_ has joined #openstack-infra20:37
*** baoli_ has joined #openstack-infra20:38
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: rename script for making stable branches
*** vivekd has quit IRC20:40
*** cloudtrainme has joined #openstack-infra20:40
*** vivekd_ is now known as vivekd20:40
*** ZZelle_ has joined #openstack-infra20:41
*** cloudtrainme has quit IRC20:44
anteayatime for me to get a walk in20:45
dimsah was just about to ping you :)20:46
dimswill do when you get back anteaya :)20:46
clarkbfungi: I think the only outstanding branch delete requests are those for murano which I called out on that thread20:46
clarkbfungi: between your handling of manuals and my grabbing of the others I think we are good20:46
fungiclarkb: thanks for taking those!20:47
anteayadims: okay thanks20:47
*** nadya has joined #openstack-infra20:47
anteayadims: be about an hour20:47
*** nadya is now known as Guest1754020:47
*** dizquierdo has joined #openstack-infra20:47
*** hashar has joined #openstack-infra20:48
*** rossella_ has quit IRC20:50
*** cloudtrainme has joined #openstack-infra20:50
*** dims_ has joined #openstack-infra20:51
*** dims has quit IRC20:52
*** mrmartin has joined #openstack-infra20:52
*** Sukhdev has quit IRC20:54
*** mrmartin has quit IRC20:54
*** hichihara has joined #openstack-infra20:56
*** ihrachys has joined #openstack-infra20:56
openstackgerritTristan Cacqueray proposed openstack-infra/shade: Add environment_files to stack_create
*** cloudtrainme has quit IRC20:57
*** dizquierdo has quit IRC20:57
*** annegentle has quit IRC20:58
openstackgerritMerged openstack-infra/system-config: Add mirror-update to cacti
*** annegentle has joined #openstack-infra20:59
*** armax has joined #openstack-infra20:59
openstackgerritMerged openstack-infra/puppet-gerrit: Tell puppet to check for cgid instead of cgi
*** salv-orlando has joined #openstack-infra21:01
*** cloudtrainme has joined #openstack-infra21:01
*** annegentle has quit IRC21:04
*** reed_ has joined #openstack-infra21:05
*** cloudtrainme has quit IRC21:06
*** dtardivel has quit IRC21:08
*** kzaitsev_mb has joined #openstack-infra21:08
*** savihou has quit IRC21:09
openstackgerritMerged openstack-infra/devstack-gate: Set DEVSTACK_GATE_TIMEOUT_BUFFER to 10
*** eil397 has joined #openstack-infra21:10
*** cloudtrainme has joined #openstack-infra21:10
*** cloudtrainme has quit IRC21:10
*** ddecapit has joined #openstack-infra21:10
*** ddecapit is now known as DuaneDeC721:10
openstackgerritMerged openstack-infra/puppet-lodgeit: Use modern template syntax
*** fitoduarte has quit IRC21:12
*** eil397 has quit IRC21:15
*** kgiusti has left #openstack-infra21:16
*** cloudtrainme has joined #openstack-infra21:16
*** eil397 has joined #openstack-infra21:17
krotscheckfungi: Sure, just a sec21:18
*** Guest17540 has quit IRC21:19
onovydhellmann: hi, can we talk about ?21:19
fungier, wrong window21:20
dhellmannonovy : sure, let's move to #openstack-release21:20
*** jpeeler has quit IRC21:20
krotscheckfungi: Looks empty to mee!21:21
*** jamespd has left #openstack-infra21:23
clarkbfungi: is there a tl;dr on where we are for using non bare images for unittests? is it a matter of adding a test to all unittest jobs to install the global bindep list and switch the image label at the same time?21:23
clarkbfungi: then next step would be to use in repo bindep lists?21:23
clarkbfungi: I wnt to try helping with that next since rax is the only place able to run unittests ucrrently21:23
*** derekh_afk is now known as derekh21:24
*** dims has joined #openstack-infra21:26
clarkbalso we should take the user survey21:26
mroddenAJaeger, fyi bashate 0.4.0 out now... hopefully helps with the pbr issues21:27
mroddeni dont see haypo on, he was asking for that earlier today21:27
*** bpokorny_ has joined #openstack-infra21:28
openstackgerritMerged openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu
clarkbovh gra image uploads take the longest, which makes sense considering the trans atlantic flight involved21:28
*** infra-red has quit IRC21:28
clarkbwe can probably work around that by running a builder in ovh gra and removing that location from the config on the current nodepool host21:29
*** dims_ has quit IRC21:29
*** jsavak has quit IRC21:30
clarkbbut then we would not be using the same image across the board, also not a huge issue with the current number of images21:31
*** bpokorny has quit IRC21:31
*** jsavak has joined #openstack-infra21:31
*** cloudtrainme has quit IRC21:32
fungiclarkb: yeah, if dib-based ubuntu-trusty images are uploading to rax consistently now, we can proceed with deciding how to cut over jobs from bare-trusty to it21:33
clarkboh right that21:33
clarkbI do not think we are consistently uploading to rax yet because the token expiration problem is still a problem21:34
clarkbI guess I should work on that some more21:34
*** mtanino has quit IRC21:34
fungiclarkb: alternatively, we can redo the tox-based jobs to run through all the steps to install missing packages and perform database configuration (those should all be no-ops on bare-trusty anyway) and then switch those jobs to use node: bare-trusty || ubuntu-trusty21:35
clarkbwell we should fix the swift upload token thing regardless so I can context switch that back in21:35
fungibut it would be simpler if we just had ubuntu-trusty images on rax first21:35
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Adding broadview-lib project and jobs.
*** jsavak has quit IRC21:36
*** cloudtrainme has joined #openstack-infra21:36
*** jsavak has joined #openstack-infra21:37
*** annegentle has joined #openstack-infra21:37
fungiclarkb: problem is so far i've really only tested the proposed job configuration change on gate-nova-python27 so either we need to broaden that experimental significantly _or_ just expect there to be possibly outlying jobs which break for various unforeseen reasons21:38
*** jaosorior has quit IRC21:38
clarkbit shouldn't be too hard to sanity check a wider range of projects21:39
fungiyep, can turn the nova job into a template and add it to a bunch of projects21:39
clarkband that can happen while we figure out uploads (I will upload new ubuntu-trusty images now so we have something recent to test on)21:40
fungii'll see if i can do that today or tomorrow and then upload empty dnm changes to the same set of projects21:40
fungifrom a transition perspective, i guess our two options are to either directly modify the existing tox-based templates or come up with some parallel versions of them21:41
fungithe former is easier and less churn, though the latter allows for a (messy) incremental transition project-by-project21:41
clarkbI think we go with updating the existing jobs, and just do it when projects aren't in the middle of feature freeze or similar21:42
fungimy preference as well21:42
clarkband start advertising real soon now what the jobs will expect for repo local bindep dep lists21:43
clarkbso that projects can work on getting those in21:43
clarkb(ubuntu-trusty is building now and rax uploads should work because I restarted services not that long ago21:43
*** cloudtrainme has quit IRC21:45
mgagneis there a way to restart the import process of an upstream project in jeepyb? Project is created fine in Gerrit and github but it didn't clone the repo21:46
*** cloudtrainme has joined #openstack-infra21:47
clarkbmgagne: typically no, since imports are destructive to any existing projects we don't run them if the project exists21:48
clarkbmgagne: what you can do is manually force push into gerritthough21:48
mgagneclarkb so it's just a matter of recloing the project myself at the location expected by gerrit?21:49
*** Sukhdev has joined #openstack-infra21:49
fungiclarkb: well, project-specific bindep lists aren't mandatory in my current design, however they are self-testing and will lead to a (slight) increase in efficiency and also we'll likely early on (if not immediately) want to stop accepting additions to the fallback bindep list21:49
*** sshnaidm has quit IRC21:49
clarkbmgagne: sort of, gerrit creates extra stuff that you don't want to just delete so what we do is force push into the gerrit remote21:49
*** |-paul-| has quit IRC21:49
clarkbmgagne: you can see how jeepyb does it, its a simple git push --force +refs/heads/*:refs/heads type deal iirc21:50
fungiright, that's basically what manage-projects does as well21:50
fungiyeah, that21:50
clarkbif you don't do that then you lose the refs/meta/config that gerrit creates by dfeault21:50
*** exploreshaifali has quit IRC21:50
mgagneoh, almost forgot about that one21:51
*** ihrachys has quit IRC21:51
mgagnewill read sources21:51
*** ihrachys has joined #openstack-infra21:51
*** kzaitsev_mb has quit IRC21:52
clarkbwe could probably do something in jeepyb that detected a half setup project and repushed21:52
clarkbbasically check if HEAD is 0000000 or the add .gitreview commit21:53
*** kzaitsev_mb has joined #openstack-infra21:55
openstackgerritDina Belova proposed openstack/requirements: Make osprofiler requirements >1.0.0
*** krtaylor has quit IRC21:55
Sam-I-Amis there a problem with
Sam-I-Amgetting clone timeoutes21:56
clarkbSam-I-Am: which protocol?21:57
anteayadims: back21:57
Sam-I-Amclarkb: whatever devstack uses... looks like git://21:58
clarkbgrafana at least doesn't report errors21:58
*** boris-42 has joined #openstack-infra21:58
clarkband the total number of connections should be manageable21:58
Sam-I-Amhmm :/21:59
clarkbSam-I-Am: if you have a repo and or source IP I can try to look at specific logs21:59
Sam-I-Amclarkb: seems to be fairly random as to what repo fails. source ip is probably nat'd somewhere. :/22:01
*** DuaneDeC7 has quit IRC22:01
*** regXboi has quit IRC22:01
*** jsavak has quit IRC22:01
greghaynesclarkb: were you able to poke at that test fail? I have a few mins now22:03
Sam-I-Amthats all i see22:03
*** trown has quit IRC22:03
*** jsavak has joined #openstack-infra22:03
clarkbgreghaynes: I havne't trying to catch up on image builds of various things22:03
greghaynesclarkb: ok, I'll have a look22:03
*** e0ne has quit IRC22:03
*** daemontool_ has joined #openstack-infra22:04
clarkbSam-I-Am: that implies tcp is failing22:04
*** akshai has quit IRC22:05
*** aysyd has quit IRC22:05
Sam-I-Amclarkb: yeah22:05
*** tzn has joined #openstack-infra22:05
mordredjeblair: ok. I'm doing a vos release on mirror.apt before doing anything else22:06
*** daemontool has quit IRC22:06
*** derekh has quit IRC22:07
*** krtaylor has joined #openstack-infra22:07
*** rockyg has joined #openstack-infra22:07
jeblairmordred: ok; has much changed since the last?22:07
mordredjeblair: don't _think_ so - but I have no idea what the volume rename actually does, so I figured I'd do one just to be sure22:08
mordred(I think it will wind up not have any bearing on this process)22:08
jeblairmordred: ok, was asking in case you wanted to run it with localauth in screen, but if not too much has changed you can probably just run locally22:08
clarkbSam-I-Am: possible route problems between you and git.o.o? or other connectivity trouble?22:08
mordredoh - nah, I ran one earlier today already22:08
jeblairor, remotely as it wore :)22:08
Sam-I-Amclarkb: other stuff seems fine. who knows :/22:09
Sam-I-Amclarkb: maybe its telling me to get away from the computer for a while22:09
mordredjeblair: mordred@mirror-update:~$ vos rename mirror.apt mirror.ubuntu22:10
Sam-I-Amwhich is a great idea22:10
mordredRenamed volume mirror.apt to mirror.ubuntu22:10
mordredjeblair: so that worked22:10
mordredthe next step is fs rmmount and fs mkmount22:10
*** jgriffith is now known as jgriffith_away22:10
*** jsavak has quit IRC22:10
mordredjeblair: do I do fs mkmount /afs/ -rw ?22:11
jeblairmordred: omit -rw22:11
*** kzaitsev_mb has quit IRC22:12
mordredjeblair: k22:12
mordredjeblair: the readonly location does not exist now22:12
*** jaosorior has joined #openstack-infra22:12
mordredmordred@mirror-update:~$ fs mkmount /afs/ mirror.ubuntu22:12
mordredmordred@mirror-update:~$ ls /afs/
mordreddb  dists  lists  pool22:12
mordredmordred@mirror-update:~$ ls /afs/
mordredls: cannot access /afs/ No such file or directory22:13
jeblairmordred: i believe if you included -rw, it would specify that in the mount and we'd never hit the ro version22:13
*** thorst has quit IRC22:13
jeblairmordred: vos release mirror22:13
jeblairmordred: the mount itself is on a volume with a read only replica too22:13
mordredthat worked22:13
*** akshai has joined #openstack-infra22:13
mordredI'm going to try running the mirror script now22:14
*** weshay has quit IRC22:14
openstackgerritMerged openstack-infra/puppet-infracloud: Expose br_name in controller and compute classes
*** thorst has joined #openstack-infra22:15
*** ihrachys has quit IRC22:16
*** hichihara has quit IRC22:16
*** jsavak has joined #openstack-infra22:16
mordredjeblair: oh - I need to not direct k5start output to the file22:16
*** jgriffith_away is now known as jgriffith22:17
clarkbgra upload is still going almost 2 hours later :/22:18
*** imcsk8|afk is now known as imcsk822:18
*** jsavak has quit IRC22:19
greghaynesclarkb: looks like its at about 20mbit22:19
openstackgerritMonty Taylor proposed openstack-infra/system-config: Suppress k5start banner when running
mordredjeblair: ^^22:20
*** thorst has quit IRC22:20
*** bpokorny_ has quit IRC22:20
*** bpokorny has joined #openstack-infra22:21
*** amitgandhinz has quit IRC22:22
*** sshnaidm has joined #openstack-infra22:22
*** esker has quit IRC22:22
*** kzaitsev_mb has joined #openstack-infra22:22
*** kzaitsev_mb has quit IRC22:22
*** kzaitsev1mb has joined #openstack-infra22:22
*** esker has joined #openstack-infra22:23
mordredjeblair: I'm currently doing a vos release on the mirror after having cleaned out multiverse and restricted22:23
kzaitsev1mbclarkb: fungi: about that murano branch delete request — I saw your email, hadn't had my hands on the client yet. Will certainly tag the branch this week and ping you on the ML about that22:23
mordredjeblair: but everything else seems to be working22:24
clarkbkzaitsev1mb: ok22:24
*** kzaitsev1mb is now known as kzaitsev_mb22:24
*** krtaylor has quit IRC22:24
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Update doc publishing spec for AFS
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Update docs-publishing spec to catch up to reality
*** jsavak has joined #openstack-infra22:25
jeblairmordred: can dumpunreferenced take a filename argument?22:25
*** esp has joined #openstack-infra22:25
*** NobodyCa1 has joined #openstack-infra22:25
jeblairmordred: if not, maybe we can have k5start run that in a shell so we still get the k5start output22:25
*** sshnaidm has quit IRC22:26
*** jsavak has quit IRC22:26
mordredjeblair: -q just suppresses the banner - but let me check22:26
mordredI do not htin kso22:26
jeblairmordred: like "k5start -- bash -c 'reprepro --confdir $REPREPRO_CONFIG dumpunfereferenced $UNREF_FILE'"22:26
*** weshay has joined #openstack-infra22:26
*** tcammann_ has joined #openstack-infra22:27
mordredjeblair: nope22:27
mordredjeblair: just dumps to stdout22:27
mordredjeblair: we could maybe "k5start -- bash -c 'reprepro --confdir $REPREPRO_CONFIG dumpunfereferenced >$UNREF_FILE'"22:27
jeblairyeah, i meant > :)22:27
*** weshay has quit IRC22:28
jeblairmordred: i'd like to avoid having the k5start output go into a blackhole for troubleshooting purposes22:28
*** weshay has joined #openstack-infra22:28
*** weshay has quit IRC22:29
*** kzaitsev_mb has quit IRC22:29
*** annegentle has quit IRC22:29
openstackgerritMonty Taylor proposed openstack-infra/system-config: Suppress k5start banner when running
mordredjeblair: ^^ how about that?22:30
jeblairmordred: commit message is wrong :)22:30
clarkbfungi: ubuntu-trusty is uploading across the board, should work against rax this first time as tokens haven't had a chance to expire but we shall see22:30
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: DNM - test out afs mirrors
ianwmrodden : i rebased if you want to take a look22:31
*** tcammann_ has quit IRC22:31
*** tzn has quit IRC22:31
*** dims has quit IRC22:31
*** esp has quit IRC22:32
mroddenianw, will do22:32
*** tzn has joined #openstack-infra22:33
*** weshay_xchat has joined #openstack-infra22:34
*** NobodyCa1 has quit IRC22:35
*** nelsnelson has quit IRC22:36
*** krtaylor has joined #openstack-infra22:36
*** tzn has quit IRC22:37
openstackgerritMonty Taylor proposed openstack-infra/system-config: Don't write k5start banner to file list
openstackgerritMonty Taylor proposed openstack-infra/project-config: Update configure_mirror to use per-region APT mirrors
mordredjeblair: ok. I think that's the stack22:38
mordredonce the system-config patch lands, we can release the locks22:38
mordredthen watch the d-g change to make sure it's all good22:38
mordredand when we're happy land the project-config patch and then rebuild images22:39
jeblairmordred: yep22:39
*** thorst has joined #openstack-infra22:40
jeblairfungi: here's a puzzle...22:41
jeblairi haven't gotten any foundation@lists.o.o mail since december...22:41
*** doug-fish has quit IRC22:41
*** sshnaidm has joined #openstack-infra22:43
jeblairi am apparently not subscribed22:43
mordredjeblair: maybe you're not really in the foundation22:44
nibalizercrinkle: mordred so omfracloud22:44
*** doug-fish has joined #openstack-infra22:44
*** thorst has quit IRC22:44
nibalizerwe have the inventory in place on the bastion host (puppetmaster)22:45
ianwdo any ruby people know why i'd be getting "/home/iwienand/.gem/ruby/gems/puppet-3.8.5/lib/puppet/vendor/safe_yaml/lib/safe_yaml/syck_node_monkeypatch.rb:42:in `<top (required)>': uninitialized constant Syck (NameError)" ?22:45
nibalizerand the ssh keys have been accepted22:45
fungijeblair: did you maybe get bounce disabled somehow?22:45
nibalizerand all the infracloud groups are children of disabled22:45
ianwit seems i've hosed my ruby installs, and it's broken puppet, rspec testing, everything :(22:45
nibalizerso in theory we can just land the patches to add the node definitions to site.pp, and un-disable the groups22:46
nibalizerpossibly running ansible-puppet manually between those two steps22:46
*** doug-fis_ has joined #openstack-infra22:46
fungijeblair: i can't think of any changes we've made that would impact the subscription list for that22:46
fungijeblair: closest i'm aware of was when we renamed the private foundation-board ml and created a new public one22:47
fungibut we didn't touch foundation@22:47
fungi(certainly not intentionally anyway)22:47
openstackgerritEmilien Macchi proposed openstack-infra/project-config: run Puppet OpenStack integration jobs on puppet-ceph
crinklenibalizer: so now depends on , so i don't know if i should make an intermediate patch so 209698 can go in right away or if you want to make 276375 go22:48
*** jpr has quit IRC22:48
*** doug-fish has quit IRC22:48
clarkbdid it auto unsubscribe you for some reason?22:49
clarkbthis is why I left gmail behind for our lists I kept getting unsubed due to fialed deliveries22:49
clarkbbut foundation@ is low traffic I would expect that from -dev22:49
*** annegentle has joined #openstack-infra22:49
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Use SSL for rabbitmq traffic
jeblairclarkb: i don't see anything in the logs about me being unsubscribed...22:49
*** doug-fis_ has quit IRC22:50
*** [1]Thelo has joined #openstack-infra22:51
*** dims_ has joined #openstack-infra22:51
jeblairindeed, mailman says i am not subscribed...22:52
fungii wonder if a moderator slipped up22:53
*** ayoung has quit IRC22:53
jeblairi wonder if i was subscribed as jeblair@o.o?22:53
fungier, list admin22:53
fungioh, perhaps22:53
*** dimtruck is now known as zz_dimtruck22:53
*** annegentle has quit IRC22:53
*** Thelo has quit IRC22:53
*** [1]Thelo is now known as Thelo22:53
jeblairif i ask for a pw reminder to that address, it does not show up in the mischief log, suggesting it really is subscribed22:53
jeblairbut i have not received the password reminder22:54
nibalizercrinkle: my sense is that 276375 is in the 'nice to have' and not 'critical path', is that accurate?22:55
crinklenibalizer: yes22:55
*** weshay_xchat has quit IRC22:55
nibalizeralso that 276375 is a state that is not currently configured22:55
crinklenibalizer: right22:55
openstackgerritIan Wienand proposed openstack-infra/puppet-diskimage_builder: Add a use_git flag
nibalizerokay so my big goal is that if we can get the infracloud hosts under CD from ansible-puppet then we can use our more traditional workflow to roll changes like 276375 out22:56
nibalizerso hopefully if we put it off for now, it will be straightforward to do later, if that makes any sense22:56
crinklenibalizer: okay i'm gonna split up 209698 so it can be merged whenever22:56
*** annegentle has joined #openstack-infra22:57
nibalizerok sweet22:57
fungijeblair: that's got me wondering if whoever manages the mailboxes has disabled non-staff addresses recently22:58
*** cloudtrainme has quit IRC22:58
fungii guess jbryce is managing those?22:58
*** dprince has quit IRC22:59
*** doug-fish has joined #openstack-infra22:59
*** cloudtrainme has joined #openstack-infra23:00
fungino clue, just trying to think of plausible explanations23:00
jeblair  jeblair@openstack.org23:00
jeblair    SMTP error from remote mail server after RCPT TO:<>:23:00
jeblair    host []: 550 5.7.1 <>:23:00
jeblair    Relay access denied.23:00
jeblairwas anyone going to tell me?23:01
jeblairi mean, i _regularly_ got mail at that address23:01
jeblairon account of the fact that i'm still part of the community23:01
*** tcammann_ has joined #openstack-infra23:01
jeblairfungi: you event sent a registration code to that address :)23:02
jeblairour account is tied to that address23:03
jeblair(because they required identity verification to set it up)23:03
*** esp has joined #openstack-infra23:03
slogan621Any other comments on  ?? Best I can tell, it looks good and I gave it a +1. Again, the understanding is we can modify this at a later time to add pypi jobs (which are not important right now and there is no pypi account yet created)23:03
jeblairrackspace notifications went to that address23:03
*** doug-fish has quit IRC23:03
*** eharney has quit IRC23:04
*** doug-fish has joined #openstack-infra23:04
jeblairwhich explains why i didn't see anything during the trouble we had recently23:04
*** jsavak has joined #openstack-infra23:04
*** sigmavirus24 is now known as sigmavirus24_awa23:04
*** mtanino has joined #openstack-infra23:06
nibalizerinfra-root taking the hiera lock23:06
*** jaosorior has quit IRC23:06
*** tcammann_ has quit IRC23:07
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node
crinklenibalizer: ^ also stacked them so they don't conflict23:07
*** dingyichen has joined #openstack-infra23:07
*** daemontool_ has quit IRC23:07
*** esp has quit IRC23:07
nibalizerinfra-root lock released23:08
nibalizercrinkle: thanks23:08
*** ayoung has joined #openstack-infra23:09
openstackgerritMerged openstack/requirements: Updated from generate-constraints
nibalizercrinkle: do you want to checkout the tip of that locally on the controller and make sure it applies correctly?23:09
openstackgerritColleen Murphy proposed openstack-infra/system-config: Use SSL for rabbitmq
*** annegentle has quit IRC23:10
crinklenibalizer: okay23:10
*** jsavak has quit IRC23:10
*** annegentle has joined #openstack-infra23:11
fungijeblair: yeah, i would have sent a registration code to every address you have in gerrit23:12
*** vivekd has quit IRC23:12
*** doug-fish has quit IRC23:14
*** doug-fish has joined #openstack-infra23:14
*** dims_ has quit IRC23:14
*** salv-orlando has quit IRC23:14
*** doug-fish has quit IRC23:15
*** doug-fish has joined #openstack-infra23:15
*** salv-orlando has joined #openstack-infra23:15
jeblairfungi: i'm having trouble communicating via irc how distressing i find the idea that the foundation would do something like that without any prior notice, or any of its technical staff knowing about it.23:16
*** annegent_ has joined #openstack-infra23:16
fungijeblair: yep, i've just tried to raise some people about this as well to get clarification23:16
fungias far as i know i don't have access to the interface which manages those addresses to check the status of them23:17
fungithat i can find anyway23:18
jeblairfungi: i do.23:18
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node
jeblairfungi: i do not have an account.23:18
jeblairfungi: nor do i have an alias.23:18
clarkb may also be similra to the thing Sam-I-Am mentioned earlier23:18
clarkbgit remote update took so long the job timed out23:18
nibalizercrinkle: is that the diff between ssl-rabbit and not-ssl-rabbit?23:18
jeblairfungi: (of course, i only use it to set up infra-root aliases for us)23:19
jeblairfungi: i don't feel like i have permission to create an alias for myself, so i will not do that.23:19
nibalizeri'll have you know, i read some of my rootermail23:19
*** annegentle has quit IRC23:19
crinklenibalizer: which that?23:19
nibalizerthe port change?23:19
crinklenibalizer: yes23:19
nibalizerim glad you're on that because it greek to me23:19
*** dims_ has joined #openstack-infra23:19
nibalizerwould you say we are 'ready' ?23:19
fungistepping out to grab a bite to eat, but will be back shortly23:20
nibalizerclarkb: jeblair fungi could you review 209698 and 213980 please23:20
*** ajmiller_ has quit IRC23:20
crinklenibalizer: i'm analyizng --noop output23:21
*** annegent_ has quit IRC23:21
*** bpokorny has quit IRC23:21
crinklenibalizer: i think this is good23:22
clarkbnibalizer: crinkle any reason to not rm and put that class directly in site.pp?23:22
*** bpokorny has joined #openstack-infra23:22
nibalizerclarkb: i need that for a follow on patch23:23
nibalizerwhich I should be rebasing23:23
*** nelsnelson has joined #openstack-infra23:23
crinkleclarkb: it started out that way, putting it in openstack_project adds more room to do things like realize admin users and other site specific things23:23
clarkbnibalizer: you need the distinct class and .pp file?23:23
clarkbcrinkle: gotcha23:23
nibalizerto add crinkle greghaynes and rcarrillocruz as infra-cloud rooters, which may not land23:23
nibalizerbut these people are allready rooting on these servers with the 'baremetal' user so that would just be a formaility23:23
*** doug-fish has quit IRC23:23
clarkbI think you need to edit the groups.txt file too23:24
clarkbbut otherwise that lgtm23:24
clarkbmordred: ^ you clarified that for jeblair recently iirc23:24
*** gordc has quit IRC23:24
nibalizerclarkb: no the groups.txt only serves to groupify otherwise unstructured openstack hosts23:24
clarkbnibalizer: jeblair had to do it for afsadmin hosts23:24
nibalizeryes those are in openstack23:24
clarkband since you have it set to group infracloud I believe you need to double account in groups.txt23:24
nibalizeri have this23:25
nibalizeri think i need a patch there though23:25
clarkbright that doesn't specify infracloud anywhere23:25
nibalizersince I got fancy23:25
nibalizeryes we/me let the groups not line up23:25
nibalizersec I will patch23:25
mgagnelooks like if dib build fails in nodepool, you can't delete it, CLI jams :-/23:26
*** doug-fish has joined #openstack-infra23:26
clarkbmgagne: you shouldn't need to delete them23:26
clarkbif build fails builder cleans up after itself and the record in the db should be removed23:27
*** doug-fish has quit IRC23:27
mgagneclarkb mine got stucked for 6 days and nodepool didn't build a new one since23:27
clarkbhowever there were some bugs around that so you may need very recent nodepool for that to work properly23:27
clarkbmgagne: yup that bug was fixed a few hours ago23:27
mgagnewill update now and test23:27
*** zz_dimtruck is now known as dimtruck23:27
clarkbthe issue was failures were not accounted by the code that checked if all the jobs were done running23:27
mgagneyou're the man23:27
clarkbso the cron never fired again23:27
*** doug-fish has joined #openstack-infra23:28
*** dimtruck is now known as zz_dimtruck23:28
openstackgerritSpencer Krum proposed openstack-infra/system-config: Collapse groups into one group
nibalizercrinkle: clarkb sorry about that23:28
*** ajmiller has joined #openstack-infra23:29
nibalizerwith that patch and crinkles patches: the group in ansible is 'infracloud' the file in hiera is 'infracloud.yaml' and variable in puppet is '$group=infracloud'23:29
mgagnecan't wait to migrate our first project to gerrit infra =)23:30
nibalizermgagne: *thumbsup*23:30
clarkbmgagne: you may need to manually delete from dib_image where state = 1; in the db to correct that broken build23:30
mgagneyea, I learned about that trick =)23:31
crinklenibalizer: i ran without --noop on the controller, seems to have survived23:31
nibalizercrinkle: awesome23:32
clarkbbut once these bugs are worked out a failed build like that should clean itself up then the next day get retried23:32
mgagnerestarting nodepool auto deleted the failed build23:32
mgagneand I'm able to delete old images too23:32
clarkbnibalizer: crinkle should depend on 277609?23:33
clarkbnibalizer: I am happy to +2 both if you want to make sure they go in the right order without another patchset23:33
crinkleclarkb: i can add depends-on23:34
*** tzn has joined #openstack-infra23:34
nibalizerit actually doesn't really matter since once we land all this no automated processes kick off23:34
nibalizerso my thinking was to get it landed23:34
nibalizerthen do ansible runs manually from the puppetmaster and see what explodes23:34
clarkbnibalizer: doesn't that group file make automated process work?23:34
clarkbor do we hae to pass an epxlicit -i inventory?23:34
nibalizerno beacuse infracloud is under [disabled:children]23:34
nibalizerso it gets caught by the !disabled guards on all our actual do_it scripts23:35
nibalizerwhich is actuall run_all.sh23:35
*** piet has quit IRC23:35
clarkband thats true even with the mismatched names because that happens before any hieraing happens23:35
clarkbnibalizer: crinkle I will approve 209698 then23:36
nibalizerclarkb: ty23:36
rm_workerr, anyone know what happened here? looks like a timeout??23:36
rm_workis it possible to just retry a gate check? or need to -A and +A?23:37
clarkbrm_work: yes I linked htat earlier, git remote update hung and the job timed out23:37
clarkbrm_work: you can just recheck it23:37
*** ccrouch has quit IRC23:38
*** doug-fish has quit IRC23:38
*** piet has joined #openstack-infra23:38
clarkbnibalizer: crinkle thta one and its child for the compute hosts has been approved23:38
clarkband +2 on the group rename change23:39
nibalizerawesome thanks23:39
nibalizerso we need one more on  27760923:39
nibalizermaybe when fungi gets back23:39
nibalizeror jeblair maybe ^?23:39
*** ccrouch has joined #openstack-infra23:39
nibalizerdims_: seems to have gotten 'e'
nibalizeroh no23:40
nibalizerdims_: is a reviewer, openstack proposal bot strikes again23:40
*** tzn has quit IRC23:40
*** doug-fish has joined #openstack-infra23:40
*** esker has quit IRC23:43
*** piet has quit IRC23:43
*** denisra has quit IRC23:44
dims_nibalizer what's up?23:44
nibalizerdims_: not much. I just mis-read a thing. Sorry to ping you23:44
nibalizerthat review is the numebr most closely matching the constant 'e'23:44
*** denisra has joined #openstack-infra23:44
dims_no worries is the eventlet bump breaking something?23:45
nibalizerno, this is entirely social chatter :)23:46
*** doug-fish has quit IRC23:46
nibalizeror casual, or whatever23:46
*** ybathia has joined #openstack-infra23:46
dims_nibalizer :)23:47
*** dkranz has quit IRC23:47
openstackgerritMerged openstack/requirements: Add manila-image-elements to project list
*** notnownikki has quit IRC23:49
openstackgerritMerged openstack/requirements: Bump ryu to 3.30
openstackgerritMerged openstack/requirements: Add sahara-scenario to projects list
*** doug-fish has joined #openstack-infra23:50
*** doug-fish has quit IRC23:50
*** doug-fish has joined #openstack-infra23:50
*** cloudtrainme has quit IRC23:51
openstackgerritMerged openstack/requirements: Add kuryr to projects.txt
openstackgerritMerged openstack/requirements: Add neutron-lbaas-dashboard to projects.txt
*** doug-fish has quit IRC23:54
*** bpokorny_ has joined #openstack-infra23:55
openstackgerritMerged openstack/requirements: For API reference doc builds, fairy-slipper 0.1.0
*** pahuang_ has quit IRC23:56
*** ZZelle_ has quit IRC23:57
*** bpokorny has quit IRC23:58
*** akshai has quit IRC23:58
dims_anteaya : got +1's from both armax and dougwig in
dims_anteaya : so can you please bless that when you get a chance23:59

Generated by 2.14.0 by Marius Gedminas - find it at!