Monday, 2015-04-27

*** shashankhegde has joined #openstack-infra00:14
*** shashankhegde has quit IRC00:20
*** baoli has joined #openstack-infra00:20
*** pcrews has joined #openstack-infra00:25
*** dims_ has quit IRC00:34
*** mahito has joined #openstack-infra00:35
*** yamada-h has joined #openstack-infra00:37
*** zhiwei has joined #openstack-infra00:39
*** abramley has quit IRC00:39
*** yamada-h has quit IRC00:41
*** salv-orlando has joined #openstack-infra00:43
*** abramley has joined #openstack-infra00:45
* lifeless looks around for pbr cores00:49
*** salv-orlando has quit IRC00:50
*** hichihara has joined #openstack-infra00:51
lifelesswho are working today00:52
*** sputnik13 has quit IRC00:53
lifeless mikal jhesketh  <- bottom of a green stack of all innocuous patches making pip firedrills less likely00:54
*** jaypipes has joined #openstack-infra00:56
jheskethlifeless: thanks, I'll take a look00:56
lifelesswell, mostly green top two aren't, but any assists on the lower end would be grand00:57
*** stevemar has quit IRC00:57
*** ddieterl_ has quit IRC01:01
mordredlifeless: I'm assuming that PBR_PIP_VERSION and PIPVERSION are intended to be not the same01:04
*** ddieterly has joined #openstack-infra01:04
mordredlifeless: (It reads like it does - but just making sure)01:05
*** ddieterl_ has joined #openstack-infra01:05
lifelessmordred: thats correct01:05
lifelessmordred: one says the one to test with and is documented as a parameter in the next patch01:05
lifelessmordred: the other is used in the mkvenv function, and is altered within the scrpt01:05
lifelesshmm pip verbose  really is verbose. I shall fix that next week01:07
jheskethlifeless: I have some nits on, but I don't want to block due to them01:08
jheskethdo you want to fix them, or shall I approve?01:08
lifelesslet me have a look01:08
*** ddieterly has quit IRC01:08
mordredlifeless: on 177503 - you've verified that that doesn't stop-on-first-failure?01:09
lifelessmordred: I plan to go through the log with a fine tooth comb01:09
lifelessmordred: (not that it matters, we weren't using the wheels *anyway*01:09
mordredlifeless: so ... the other thing I just rhough of of why we did that01:09
*** achanda has joined #openstack-infra01:09
mordreda single pass will get you one wheel for each requirement01:10
mordredwhereas, because of depsolver things - the while loop _may_ get you more than one version of something in the wheel house01:10
mordredwhich might be needed to satisfy different sequence combinations01:10
mordredI'm not saying this holds up01:10
mordredbut as you sift through the logs01:11
mordredsomething for your brainhole01:11
lifelessjhesketh: one I agree with, easy to fix01:11
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade
mordredworst latency way to test that ...01:13
mordredbut it works locally01:13
mordredso - you know - adding print statement and pushing to gate to see why gate is different ...01:13
lifelessmordred: yeah, its a nice theory01:14
lifelessmordred: but because we do them all in isolation what it actually means is that you'll get the wrong wheel.01:14
lifelessmordred: foo <=3.4 in global requirements -> wheel of 3.4.01:14
lifelessmordred: bar deps foo <=3.3 somewhere, 3.3 is what we end up using.01:14
lifelessmordred: eitherway, the auto wheel building in 7.0.0 will address that01:15
*** neoice_ is now known as neoice01:15
lifelessmordred: and we can rip out this wheel stuff entirely01:15
*** sandywalsh has joined #openstack-infra01:15
*** sigmavirus24_awa has quit IRC01:16
*** Trozz_ has quit IRC01:16
*** Trozz has joined #openstack-infra01:17
*** odyssey4me has quit IRC01:17
*** fungi has quit IRC01:18
*** ttx has quit IRC01:18
*** btully has joined #openstack-infra01:18
*** sandywalsh_ has quit IRC01:19
*** ctracey has quit IRC01:20
openstackgerritlifeless proposed openstack-dev/pbr: Stop testing install
openstackgerritlifeless proposed openstack-dev/pbr: Test pip install -e of projects.
openstackgerritlifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many.
*** odyssey4me has joined #openstack-infra01:20
openstackgerritlifeless proposed openstack-dev/pbr: Improve
openstackgerritlifeless proposed openstack-dev/pbr: Use /opt/git directly
lifelessmordred: how big is /opt/git?01:20
*** fungi has joined #openstack-infra01:21
*** ttx has joined #openstack-infra01:22
*** sigmavirus24_awa has joined #openstack-infra01:22
lifelessttx: hi01:22
*** ctracey has joined #openstack-infra01:22
lifelessttx: [is that really you?]01:22
*** yamamoto_ has joined #openstack-infra01:30
*** jamesmcarthur has joined #openstack-infra01:31
*** yamamoto_ has quit IRC01:32
*** sdake has joined #openstack-infra01:32
*** yamada-h has joined #openstack-infra01:33
*** yamamoto has quit IRC01:33
*** armax has joined #openstack-infra01:33
*** jyuso1 has quit IRC01:36
*** kaisers has quit IRC01:36
*** kaisers1 has quit IRC01:36
*** jyuso1 has joined #openstack-infra01:36
*** btully has quit IRC01:37
*** kaisers has joined #openstack-infra01:39
*** kaisers1 has joined #openstack-infra01:39
*** btully has joined #openstack-infra01:39
*** sdake_ has joined #openstack-infra01:48
*** reazem_ is now known as reazem01:50
*** sdake has quit IRC01:51
*** sputnik13 has joined #openstack-infra01:52
*** stevemar has joined #openstack-infra01:53
*** baoli has quit IRC01:57
*** erikmwilson_ is now known as erikmwilson01:59
*** asettle is now known as asettle-afk02:02
*** achanda has quit IRC02:02
*** mjordan has quit IRC02:02
*** jlvillal has quit IRC02:08
*** amitgandhinz has joined #openstack-infra02:09
*** jlvillal has joined #openstack-infra02:09
*** amitgandhinz has quit IRC02:17
*** amitgandhinz has joined #openstack-infra02:18
*** armax has quit IRC02:24
*** amitgandhinz has quit IRC02:25
*** yamada-h has quit IRC02:25
*** amitgandhinz has joined #openstack-infra02:26
mordredlifeless: I mean ... it's not small?02:26
mordredlifeless: but I don't know the size off the top of my head02:26
*** amitgandhinz has quit IRC02:28
*** Longgeek has quit IRC02:28
*** yamada-h has joined #openstack-infra02:31
*** Longgeek has joined #openstack-infra02:33
*** yamada-h_ has joined #openstack-infra02:33
*** baoli has joined #openstack-infra02:35
*** sdake_ has quit IRC02:35
*** sputnik13 has quit IRC02:36
*** yamada-h has quit IRC02:37
morganfainbergsdague: I'm happy to co-lead the Service Catalog Standardization cross-project thing if no one else wants to. but I'd rather have someone else lead and make sure keystone folks are there.02:40
morganfainbergsdague: but i for sure wouldn't want to be the sole leader for it.02:40
tchaypolifeless: looking at 177505, I’m not sure how this is testing pip more thoroughly - unless that just means that we check for migrate.cfg in $pipenv rather than $installenv02:43
*** salv-orlando has joined #openstack-infra02:46
*** shashankhegde has joined #openstack-infra02:51
*** jamesmcarthur has quit IRC02:51
*** sdake has joined #openstack-infra02:52
*** wuhg has joined #openstack-infra02:52
*** salv-orlando has quit IRC02:56
*** asettle-afk is now known as asettle02:59
*** salv-orlando has joined #openstack-infra03:02
*** salv-orlando has quit IRC03:04
*** woodster_ has joined #openstack-infra03:05
*** baoli has quit IRC03:08
*** otter768 has quit IRC03:20
*** armax has joined #openstack-infra03:21
*** btully has quit IRC03:22
*** armax has quit IRC03:23
*** sdake_ has joined #openstack-infra03:28
*** ddieterl_ has quit IRC03:29
*** sdake has quit IRC03:32
*** yamada-h has joined #openstack-infra03:34
*** yamada-h has quit IRC03:39
*** mjordan has joined #openstack-infra03:40
*** ildikov has quit IRC03:40
*** tjones1 has joined #openstack-infra03:45
*** yamada-h has joined #openstack-infra03:47
*** yamada-h has quit IRC03:47
*** yamada-h has joined #openstack-infra03:50
*** ildikov has joined #openstack-infra03:54
*** tjones1 has quit IRC03:57
*** jtriley has joined #openstack-infra04:01
*** sarob has joined #openstack-infra04:01
*** jtriley has quit IRC04:05
*** yamada-h has joined #openstack-infra04:10
*** yamada-h has quit IRC04:10
*** mjordan has left #openstack-infra04:16
*** Longgeek has quit IRC04:17
*** Longgeek has joined #openstack-infra04:18
*** davideagnello has joined #openstack-infra04:24
*** davideagnello has quit IRC04:28
*** sarob has quit IRC04:28
*** sarob has joined #openstack-infra04:29
*** ddieterly has joined #openstack-infra04:30
*** yamamoto has joined #openstack-infra04:31
*** ildikov has quit IRC04:32
*** sarob has quit IRC04:34
*** ddieterly has quit IRC04:34
*** yamada-h has joined #openstack-infra04:40
*** yamada-h has quit IRC04:46
*** coolsvap|afk is now known as coolsvap04:47
*** ildikov has joined #openstack-infra04:49
*** ildikov has quit IRC04:53
*** AJaeger has joined #openstack-infra04:54
*** yfried__ has quit IRC04:56
AJaegerfungi, clarkb, jeblair, pleia2, jhesketh, SergeyLukjanov: Gerrit is stuck, no no events are processed at all.04:56
AJaegerttx: if nobody can help at this time, could you send an alert, please?04:57
*** yamada-h has joined #openstack-infra04:59
*** yamada-h has quit IRC04:59
jheskethAJaeger: I'm here, but I have no powers :-(05:02
jheskethtrying to see what merged that might have stuck things05:02
AJaegerjhesketh: nothing seem to have merged in the last 24 hours but this is since 3 hours or so05:03
AJaegerat least my latest emails are from that time05:04
jheskethyeah, nothing is standing out to me05:04
jheskethlooking at graphite indicates that zuul has lost connection with gerrit05:05
*** Trozz has quit IRC05:11
jheskethAJaeger: looks like gerrit isn't streaming any events (to my zuul/user either)05:11
*** Trozz has joined #openstack-infra05:12
AJaegerso, we need somebody to restart gerrit?05:15
AJaegerSergeyLukjanov: are you around to do that?05:15
jheskethwell possibly05:16
jheskethI cna't find any other hints in cacti or graphite though05:18
*** jamespage_ has joined #openstack-infra05:19
AJaegerSo, nobody around - in that case, let me send an email to openstack-dev to inform people05:20
*** otter768 has joined #openstack-infra05:21
*** deepakcs has joined #openstack-infra05:23
* AJaeger sent mail05:26
*** otter768 has quit IRC05:26
*** jamespage_ has quit IRC05:29
*** subscope_ has joined #openstack-infra05:30
*** hdd has joined #openstack-infra05:30
*** subscope_ has quit IRC05:31
*** BharatK has joined #openstack-infra05:31
jheskethAJaeger: thanks :-)05:33
*** btully has joined #openstack-infra05:33
*** wolsen_ is now known as wolsen05:33
AJaegerI have to go now, will be back later05:34
AJaegerthanks, jhesketh for your investigation!05:35
*** AJaeger has quit IRC05:35
*** abregman has joined #openstack-infra05:35
jheskethno trouble, sorry I can't help more!05:36
*** achanda has joined #openstack-infra05:39
*** ajo has joined #openstack-infra05:43
*** masayukig has quit IRC05:44
*** sweston has quit IRC05:44
*** persia has quit IRC05:44
*** sweston has joined #openstack-infra05:45
*** masayukig has joined #openstack-infra05:46
*** persia has joined #openstack-infra05:46
*** persia has quit IRC05:46
*** persia has joined #openstack-infra05:46
*** salv-orlando has joined #openstack-infra05:46
*** cpallares has joined #openstack-infra05:47
*** [HeOS] has quit IRC05:47
*** tnovacik has joined #openstack-infra05:51
*** salv-orlando has quit IRC05:54
*** sdake_ has quit IRC05:56
*** sdake has joined #openstack-infra05:56
*** dannywilson has joined #openstack-infra05:57
*** yamada-h has joined #openstack-infra05:59
*** dannywilson has quit IRC06:02
*** dannywilson has joined #openstack-infra06:02
*** ildikov has joined #openstack-infra06:03
*** yamada-h has quit IRC06:04
*** morgabra has quit IRC06:04
*** morgabra has joined #openstack-infra06:04
*** mrunge has joined #openstack-infra06:05
*** mrunge has quit IRC06:06
*** kaisers1 has quit IRC06:06
*** kaisers has quit IRC06:06
*** Longgeek has quit IRC06:07
*** dtantsur|afk is now known as dtantsur06:07
*** Longgeek has joined #openstack-infra06:09
*** zul has joined #openstack-infra06:11
*** _nadya_ has joined #openstack-infra06:14
*** kaisers has joined #openstack-infra06:16
*** mrunge has joined #openstack-infra06:18
*** mrunge has joined #openstack-infra06:18
*** kiseok7_ has joined #openstack-infra06:19
*** _nadya_ has quit IRC06:19
*** dalgaaf has joined #openstack-infra06:20
*** masayukig has quit IRC06:21
*** afazekas has joined #openstack-infra06:21
*** vlaza has joined #openstack-infra06:22
*** liam__ has joined #openstack-infra06:22
*** liam__ has quit IRC06:22
*** masayukig has joined #openstack-infra06:23
*** dalgaaf has quit IRC06:24
*** e0ne has joined #openstack-infra06:24
*** dalgaaf has joined #openstack-infra06:24
*** dannywilson has quit IRC06:25
*** Qiming_ has joined #openstack-infra06:27
*** stevemar has quit IRC06:27
*** e0ne has quit IRC06:28
*** Sukhdev has joined #openstack-infra06:29
*** scheuran has joined #openstack-infra06:30
*** scheuran has quit IRC06:30
*** scheuran has joined #openstack-infra06:30
*** ddieterly has joined #openstack-infra06:31
*** Longgeek_ has joined #openstack-infra06:34
*** Sukhdev has quit IRC06:34
*** AJaeger has joined #openstack-infra06:35
*** ddieterly has quit IRC06:35
*** yfried__ has joined #openstack-infra06:36
*** Longgeek has quit IRC06:37
*** kaisers has quit IRC06:39
*** woodster_ has quit IRC06:40
*** zul has quit IRC06:40
*** salv-orlando has joined #openstack-infra06:42
*** Yingxin has joined #openstack-infra06:43
*** ssam2 has joined #openstack-infra06:44
*** salv-orlando has quit IRC06:44
*** Qiming has joined #openstack-infra06:46
*** ttx has quit IRC06:47
*** ttx has joined #openstack-infra06:47
*** Qiming_ has quit IRC06:48
*** AJaeger has quit IRC06:50
*** AJaeger has joined #openstack-infra06:51
*** camunoz is now known as camunoz_gone06:51
*** AJaeger has quit IRC06:53
* ttx catches up06:54
*** AJaeger has joined #openstack-infra06:54
jheskethttx: basically we just need soembody with root to investigate/punch review.o.o06:55
ttxAck, my powers are limited to giving status updates through the statusbot though06:55
ttxso, Zuul not processing events ?06:55
jheskethttx: okay, so the status is that zuul isn't receiving events from gerrit therefore nothing is queued for testing or merging06:55
*** Bsony has joined #openstack-infra06:56
AJaegerttx, so could you give status updates again, please?06:56
* AJaeger sent already an email to openstack-dev but let's spread the word06:56
AJaegergood morning, ttx!06:56
ttx#status alert Zuul is currently not processing any event, so no check / gate tests are being run. The issue is being investigated.06:58
ttxhah, no statusbot06:58
*** sdake has quit IRC06:58
ttxWe also need someone to punch the statusbot06:59
ttxOne of those days, I guess06:59
ttxlifeless: yes it is me07:00
AJaegerSo, options are: SergeyLukjanov is around and can help - unlikely, hasn't reacted yet - or waiting for fungi to wake up07:01
*** ildikov has quit IRC07:01
rcarrillocruzwe need moar infra cores/root, just sayin'...07:02
*** ivar-lazzaro has quit IRC07:02
*** yamada-h has joined #openstack-infra07:03
rcarrillocruzin EMEA that is07:04
yolandarcarrillocruz, +1 on that. Set a +1 in my topic for the sprint, please :)07:05
* jhesketh wishes he could help more07:05
*** jcoufal has joined #openstack-infra07:06
yolandawe know that system in our team from memory, we do the same as infra every day07:06
rcarrillocruzjhesketh: it must be quite late for you right?07:06
yolandaand will be glad to help07:06
rcarrillocruzwhich tz are you in?07:06
*** jcoufal is now known as 7JTAAIHSL07:06
*** jcoufal has joined #openstack-infra07:06
*** jcoufal has quit IRC07:06
*** jcoufal has joined #openstack-infra07:06
*** jcoufal has quit IRC07:06
rcarrillocruzi know AU has a handful of different TZs07:06
jheskethrcarrillocruz: nope, just past 5pm07:07
jhesketh(it would be 6pm if we were still in daylight savings)07:07
*** kaisers has joined #openstack-infra07:08
jheskethwestern australia (of which I'm not sure we have any devs from) is UTC+7, so earlier for them07:08
*** mpaolino has joined #openstack-infra07:08
StevenKI thought WA was always UTC+8, because daylight savings isn't a thing there07:09
*** ildikov has joined #openstack-infra07:09
jheskethStevenK: err, yes, sorry, my mistake07:09
*** shardy_ has joined #openstack-infra07:09
StevenKThey've experimented with it three times in the last n years, but it keeps getting shot down07:09
*** shardy has quit IRC07:11
*** hyakuhei has joined #openstack-infra07:11
rcarrillocruzopenstack is so mainstream now that i'd assume we have devs in every possible TZ these days07:11
*** gema has joined #openstack-infra07:12
*** gema has quit IRC07:12
*** gema has joined #openstack-infra07:13
*** shardy_ has quit IRC07:14
*** apmelton_away has quit IRC07:15
*** shardy has joined #openstack-infra07:15
*** apmelton has joined #openstack-infra07:16
*** Nakato has quit IRC07:16
*** Nakato has joined #openstack-infra07:16
*** freyes has quit IRC07:16
*** freyes has joined #openstack-infra07:18
*** MaxV has joined #openstack-infra07:21
lifelessttx: it was mor itming t t07:21
*** viktors has joined #openstack-infra07:22
viktorshi folks! Can somebody look  at patch (Add Ironic python3.4 experimental job) ?07:22
*** otter768 has joined #openstack-infra07:22
*** saggi has joined #openstack-infra07:24
yolandajhesketh, thx for your quick review, i amended the text according to your comments07:24
*** markvoelker has joined #openstack-infra07:26
*** otter768 has quit IRC07:27
jheskethyolanda: cool, I'll take a look later07:28
yolandai think that Clark's suggestions to remove gearman, and just execute a simple test script, simplified that spec. I'm worried about future needs , specially if we need to reuse some jobs already declared on jenkins, but i prefer keeping that simple, and iterate later if we have more needs07:29
SergeyLukjanovAJaeger, hey07:29
SergeyLukjanovAJaeger, /me just woke up :)07:29
AJaegermorning, SergeyLukjanov ! Didn't want to wake you up ;)07:30
*** markvoelker has quit IRC07:30
AJaegerSergeyLukjanov, Zuul is not processing any events, seems the gerrit queue is stuck - could you revive it again, please?07:30
SergeyLukjanovAJaeger, looking07:31
AJaegerAnd bring us back the statusbot here, plesae?07:31
AJaegerthanks, SergeyLukjanov07:31
*** ibiris_away is now known as ibiris07:31
SergeyLukjanovAJaeger, it looks like event stream is dead, yeah07:31
*** ddieterly has joined #openstack-infra07:32
SergeyLukjanovAJaeger, jhesketh , I think I'll just restart gerrit, zuul and statusbot07:34
jheskethSergeyLukjanov: if the logs aren't helpful that seems like a reasonable step07:36
*** yamada-h has quit IRC07:36
lifelesswow ok so07:36
lifeless20G in one chromium process07:36
*** ddieterly has quit IRC07:36
*** salv-orlando has joined #openstack-infra07:36
saggiAJaeger: I'm part of the dragonflow team. We want to change the submission policy to fast forward only. Is it just a matter of setting `mergeContent = false` in `project-config/gerrit/acls/stackforge/dragonflow.config`?07:37
lifelessttx: so I wanted to ask you07:37
lifelessttx: do we have a timeslot for the dependency version management session07:37
AJaegersaggi, sorry, can't help with that.07:37
saggiAJaeger: Do you happen to know who can?07:38
lifelessttx: and if so, is it against ops / 3rd-party CI ops / infra / packager sessions07:38
AJaegersaggi, I hope the other infra cores can help - please ask again during US time07:38
*** BobBall_ is now known as BobBall07:39
*** gillliard is now known as gilliard07:39
*** hyakuhei has quit IRC07:39
saggiAJaeger: They wake up so late :). We also have an issue with the GitHub mirror description not syncing. Is that someone else as well?07:39
AJaegersaggi, AFAIK github description is only synced at initial creation, so if you change it later, it won't be updated.07:40
*** jlanoux has joined #openstack-infra07:40
*** samuelBartel has joined #openstack-infra07:41
AJaegersaggi, but double check with the US based part of the team for this, please07:41
SergeyLukjanovsaggi, I think mergeContent is a bit different config, if you'd like to produce strictly linear history the submit type "Fast Forward Only" needed07:41
*** yamada-h has joined #openstack-infra07:42
SergeyLukjanovinstead of our defultt "Merge If Necessary"07:42
*** sdake has joined #openstack-infra07:42
*** sdake has quit IRC07:42
*** sdake has joined #openstack-infra07:42
ttxlifeless: we were planning to abuse the Friday Infra/QA/RelMgt sprint07:43
*** mpavone has joined #openstack-infra07:45
samuelBartelhi all!07:45
samuelBarteli have a question about the process
SergeyLukjanovtons of errors:07:45
SergeyLukjanov[2015-04-27 07:42:48,506] WARN : Exception caught07:45 Connection reset by peer07:45
SergeyLukjanovjhesketh, ^^07:45
*** Bsony has quit IRC07:46
*** yamada-h has quit IRC07:46
samuelBartelwhat is the correct workflow to be add to the gerrit groups after creation of the a new project?07:47
samuelBartelbug in launnchpad? request by email?07:47
SergeyLukjanovjhesketh, and some errors with hooks exec
AJaegersamuelBartel, Asking here or email to openstack-infra. Give the gerrit group and your email address or gerrit login. Once you've been added, you can use to add more07:48
samuelBartelAJaeger, great thank you07:48
*** hyakuhei has joined #openstack-infra07:48
samuelBartelAJaeger, can I give you the informations (group name and email address)?07:49
*** Hal has joined #openstack-infra07:49
AJaegersamuelBartel, just give it here and one of the admins will read the backlog and do it later.07:49
*** Hal is now known as Guest1129007:50
*** skolekonov has joined #openstack-infra07:50
jheskethSergeyLukjanov: hmm, is that after a restart?07:50
saggiSergeyLukjanov: Can't find any project that uses "Fast Forward Only". Is this what I'm supposed to do?
SergeyLukjanovjhesketh, nope, I'm still reading logs before restart07:51
SergeyLukjanovjhesketh, will do it for another 5 mins and do restart07:51
jheskethSergeyLukjanov: can you test the credentials that gerrit has to launchpad ?07:51
SergeyLukjanovjhesketh, I don't see any really bad stuff in the logs07:52
SergeyLukjanovjhesketh, there is only one such error in logs07:52
AJaegersaggi, since no other project is doing this, I wonder whether you really need it - if you change it, make a verbose commit message explaining the reasoning.07:52
*** arxcruz has joined #openstack-infra07:52
*** Longgeek_ has quit IRC07:54
SergeyLukjanovjhesketh, there is a branch of another exceptions in logs -
*** hashar has joined #openstack-infra07:57
*** jistr has joined #openstack-infra07:57
*** freyes has quit IRC07:59
*** rm_work has quit IRC07:59
*** rm_work|away has joined #openstack-infra07:59
*** rm_work|away is now known as rm_work07:59
*** rm_work has joined #openstack-infra07:59
*** persia has quit IRC08:00
jheskethSergeyLukjanov: hard to tell much off those pastes :-(08:00
*** andrey-mp has joined #openstack-infra08:00
*** persia has joined #openstack-infra08:00
*** persia has quit IRC08:00
*** persia has joined #openstack-infra08:00
*** zul has joined #openstack-infra08:00
SergeyLukjanovjhesketh, /me will send notice: Restarting gerrit because it stopped sending events08:01
*** freyes has joined #openstack-infra08:01
SergeyLukjanov(ETA 15 mins)08:01
jheskethSergeyLukjanov: cool08:02
AJaegerSergeyLukjanov, did you restart the statusbot so that you can send notices?08:02
*** btully has quit IRC08:02
SergeyLukjanovAJaeger, yup08:02
SergeyLukjanovAJaeger, fighting slow internet connection :)08:02
*** Longgeek has joined #openstack-infra08:03
*** shashankhegde has quit IRC08:03
*** hashar is now known as hasharMeeting08:03
*** fhubik has joined #openstack-infra08:04
*** Guest11290 has quit IRC08:05
*** flashgordon has quit IRC08:06
*** yamada-h has joined #openstack-infra08:06
*** openstackstatus has joined #openstack-infra08:06
*** ChanServ sets mode: +v openstackstatus08:06
*** zul has quit IRC08:08
SergeyLukjanov#status notice Restarting gerrit because it stopped sending events (ETA 15 mins)08:09
openstackstatusSergeyLukjanov: sending notice08:09
SergeyLukjanovjhesketh, AJaeger ^^08:09
-openstackstatus- NOTICE: Restarting gerrit because it stopped sending events (ETA 15 mins)08:09
*** persia has quit IRC08:10
*** yamada-h has quit IRC08:10
*** persia has joined #openstack-infra08:11
openstackstatusSergeyLukjanov: finished sending notice08:11
*** sergsh has joined #openstack-infra08:11
*** _nadya_ has joined #openstack-infra08:12
*** openstackgerrit has quit IRC08:13
SergeyLukjanovgoodbye gerrit08:13
*** Ala has joined #openstack-infra08:14
mkodererok, Gerrit is back again :)08:16
*** dtantsur is now known as dtantsur|brb08:16
*** achanda has quit IRC08:16
*** openstackgerrit has joined #openstack-infra08:16
jheskethSergeyLukjanov: cool, events appear to be streaming again08:17
SergeyLukjanovjhesketh, yeah, I already see firs vents08:17
SergeyLukjanovfirst events*08:17
jheskethSergeyLukjanov: all 3rd parties will probably need to restart their zuul's08:17
SergeyLukjanovjhesketh, I will send notice to -dev08:18
*** freyes has quit IRC08:18
SergeyLukjanovjhesketh, yeah, we have first jobs running in zuul - http://zuul.openstack.org08:18
jheskethSergeyLukjanov: and possibly the 3rd party ops announce list?08:18
SergeyLukjanovjhesketh, yeah08:18
AJaegerSergeyLukjanov, do we need to recheck all jobs?08:19
*** sarob has joined #openstack-infra08:19
*** derekh has joined #openstack-infra08:19
mkodererAJaeger: seems so08:19
AJaegerSergeyLukjanov, shall I send out an email?08:20
jheskethSergeyLukjanov: I have a script that gives you a set of rechecks to run08:21
*** vponomaryov has quit IRC08:21
*** freyes has joined #openstack-infra08:21
*** liam__ has joined #openstack-infra08:22
*** liam__ has quit IRC08:23
*** vponomaryov has joined #openstack-infra08:23
*** vponomaryov has left #openstack-infra08:23
*** vponomaryov has joined #openstack-infra08:23
SergeyLukjanovAJaeger, oops, I already send an email, if you'd like to add details08:23
AJaegerjhesketh, that is only nova, isn't it?08:24
AJaegerSergeyLukjanov, let me read you email and decide then;)08:24
jheskethAJaeger: should be all projects?08:24
*** sarob has quit IRC08:24
SergeyLukjanovjhesketh, I see only nova in a list ;)08:24
SergeyLukjanovAJaeger, it's a sleepy morning notice of the hell :)08:24
*** [HeOS] has joined #openstack-infra08:24
*** dguitarbite has joined #openstack-infra08:25
SergeyLukjanovjhesketh, it looks like project name is hardcoded, I checked a few of the CR ids and it's not nova08:25
jheskethplease hold08:26
jheskethah, the recheck line is nova only, but I think the changes should be everything08:27
*** markvoelker has joined #openstack-infra08:27
AJaegerSo, SergeyLukjanov do you want to run it? In that case I can edit my email and say we run rechecks08:29
jheskethSergeyLukjanov: okay, trying again:
SergeyLukjanovAJaeger, jhesketh, do you remember, does we run rechecks previously when gerrit failed?08:30
jheskethhowever that's just changes missing any votes, not any that should be running in gate08:30
SergeyLukjanovAJaeger, jhesketh honestly I don't remember08:30
jheskethSergeyLukjanov: I'm not sure what has been done sorry08:30
AJaegerI don't remember.08:30
AJaegerBut would be nice to do it...08:30
jheskethSergeyLukjanov: bash history?08:31
AJaegerIf there's a planned downtime, we normally requeue them...08:31
*** xnox has joined #openstack-infra08:31
SergeyLukjanovthere is a short list08:31
SergeyLukjanovwe could schedule it w/o any issues08:31
*** markvoelker has quit IRC08:32
SergeyLukjanovjhesketh, thx for the list, I'll run in a second08:32
*** freyes has quit IRC08:32
SergeyLukjanovAJaeger, ^^08:32
AJaegerSergeyLukjanov, ok, will send out an updated email08:33
*** ddieterly has joined #openstack-infra08:33
jheskethSergeyLukjanov: there will still be a disconnect between those and ones that should be in gate08:33
*** shardy_ has joined #openstack-infra08:33
*** shardy has quit IRC08:34
*** freyes has joined #openstack-infra08:35
SergeyLukjanovjhesketh, at least all changes will have votes08:35
AJaegerjhesketh, it covers most ;) My email states that we tried ;)08:35
jheskethI'm working on a more complete list08:35
*** Ala has quit IRC08:36
AJaegerthanks, SergeyLukjanov and jhesketh !08:36
jheskethSergeyLukjanov: the api limits my queries to 500, so it's probably missing quite a few :-(08:37
*** zhiwei has quit IRC08:37
SergeyLukjanovjhesketh, I think it's ok, thanks!08:37
openstackgerritSaggi Mizrahi proposed openstack-infra/project-config: Change dragonflow commit type
*** ddieterly has quit IRC08:37
*** Ala has joined #openstack-infra08:38
*** shardy_ has quit IRC08:38
*** zul has joined #openstack-infra08:38
*** andrey-mp has quit IRC08:38
*** fhubik is now known as fhubik_afk08:39
*** shardy has joined #openstack-infra08:39
*** zul has quit IRC08:41
*** samuelBartel has quit IRC08:41
AJaegersaggi, thanks for the verbose explanation. I'll abstain from commenting for now, others need to review this.08:41
saggiAJaeger: about GitHub description. Is there someone that can manually change it for us? Can we do it? We want the project to get traction so we try to get our info online to be helpful and accurate.08:41
*** zul has joined #openstack-infra08:41
*** ildikov has quit IRC08:44
AJaegersaggi, please ask the US based part of the team later08:44
saggiAJaeger: OK, thanks for all the help08:44
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed
openstackbug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] - Assigned to Bharat Kumar Kobagana (bharat-kobagana)08:44
*** mfmcdonagh has quit IRC08:44
*** jamespage has quit IRC08:46
*** jamespage has joined #openstack-infra08:46
*** hyakuhei has quit IRC08:50
*** zhiwei has joined #openstack-infra08:50
*** sdake_ has joined #openstack-infra08:51
*** Darkwan has quit IRC08:52
*** fhubik_afk is now known as fhubik08:54
*** notnownikki has joined #openstack-infra08:55
*** sdake has quit IRC08:55
*** ildikov has joined #openstack-infra08:56
*** achanda has joined #openstack-infra08:59
*** e0ne has joined #openstack-infra08:59
*** mahito has quit IRC09:00
*** Darkwan has joined #openstack-infra09:01
*** _nadya_ has quit IRC09:03
*** Hal has joined #openstack-infra09:03
*** Hal is now known as Guest8264909:04
jheskeththe gerrit documentation seems to lie09:04
jheskethI can't do a query with an offset09:04
* jhesketh is probably failing at something obvious09:04
*** mpavone has quit IRC09:05
*** skolekonov has quit IRC09:05
*** flashgordon has joined #openstack-infra09:06
*** dalgaaf has quit IRC09:07
jheskethhmm, it doesn't look like you can search with an offset in gerrit 2.8 :-(09:09
*** e0ne has quit IRC09:10
jheskethSergeyLukjanov: since the nodepool is exausted there is no point in figuring out the missing reviews due to the offset... I'll run my tool again later to see if there are more rechecks to be done, but for now we should catch most and others will catch the system up too09:10
jhesketh(others being people)09:10
*** sdake_ has quit IRC09:13
*** e0ne has joined #openstack-infra09:13
*** Qiming has left #openstack-infra09:14
*** sdake has joined #openstack-infra09:15
*** pelix has joined #openstack-infra09:16
*** mpavone has joined #openstack-infra09:17
*** dtantsur|brb is now known as dtantsur09:19
*** fhubik is now known as fhubik_afk09:25
*** markvoelker has joined #openstack-infra09:27
*** dmellado has quit IRC09:28
*** fhubik_afk is now known as fhubik09:29
*** teran has quit IRC09:29
*** sks has joined #openstack-infra09:29
*** alexpilotti has joined #openstack-infra09:30
*** zul has quit IRC09:30
*** jamespage_ has joined #openstack-infra09:32
*** AJaeger has quit IRC09:32
*** Longgeek has quit IRC09:33
*** zul has joined #openstack-infra09:33
*** ddieterly has joined #openstack-infra09:33
*** Longgeek has joined #openstack-infra09:34
*** rossella_ has joined #openstack-infra09:34
*** ociuhandu has joined #openstack-infra09:35
*** markvoelker has quit IRC09:36
*** ddieterly has quit IRC09:37
*** yamamoto has quit IRC09:41
openstackgerritValeriy Ponomaryov proposed openstack/requirements: Add python-manilaclient to global requirements.
*** AJaeger has joined #openstack-infra09:44
*** fhubik is now known as fhubik_afk09:44
*** fhubik_afk is now known as fhubik09:46
*** BobBall_ has joined #openstack-infra09:50
*** flip214 has quit IRC09:51
*** alexpilotti has quit IRC09:52
*** yamada-h has joined #openstack-infra09:53
*** BobBall has quit IRC09:54
*** mpavone has quit IRC09:54
*** achanda has quit IRC09:55
*** dguitarbite has quit IRC09:59
*** achanda has joined #openstack-infra10:00
*** mpavone has joined #openstack-infra10:01
*** dguitarbite has joined #openstack-infra10:01
*** yfried__ is now known as yfried|afk10:03
*** yfried|afk is now known as yfried__10:06
*** alexpilotti has joined #openstack-infra10:07
*** sks has quit IRC10:08
*** sarob has joined #openstack-infra10:08
*** achanda has quit IRC10:11
*** mrunge has quit IRC10:11
*** sarob has quit IRC10:13
*** dims has joined #openstack-infra10:14
*** Longgeek has quit IRC10:14
*** Guest82649 has quit IRC10:17
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed
openstackbug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] - Assigned to Bharat Kumar Kobagana (bharat-kobagana)10:18
*** mattymo_ is now known as mattymo10:21
*** Longgeek has joined #openstack-infra10:26
*** markvoelker has joined #openstack-infra10:28
*** hichihara has quit IRC10:30
*** samueldmq has joined #openstack-infra10:32
*** markvoelker has quit IRC10:33
*** ddieterly has joined #openstack-infra10:34
*** dalgaaf has joined #openstack-infra10:36
*** ddieterly has quit IRC10:38
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
*** yamada-h has joined #openstack-infra10:40
*** yamada-h has quit IRC10:40
*** sdake has quit IRC10:41
*** zul has quit IRC10:41
*** jamespage_ has quit IRC10:41
*** mrunge has joined #openstack-infra10:45
*** ociuhandu has quit IRC10:50
*** sdake has joined #openstack-infra10:51
*** lxsli0 is now known as lxsli10:52
*** u_glide has joined #openstack-infra10:54
*** ildikov has quit IRC10:56
*** ildikov has joined #openstack-infra10:56
*** alexpilotti has quit IRC10:58
*** fhubik is now known as fhubik_afk10:59
*** yamamoto has joined #openstack-infra10:59
*** teran has joined #openstack-infra11:00
*** pc_m has joined #openstack-infra11:00
*** yamada-h has joined #openstack-infra11:01
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
*** yamamoto has quit IRC11:02
*** yamamoto has joined #openstack-infra11:02
*** _nadya_ has joined #openstack-infra11:02
*** teran has quit IRC11:03
*** teran has joined #openstack-infra11:03
*** sks has joined #openstack-infra11:03
*** zhiwei has quit IRC11:03
*** pc_m has quit IRC11:04
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
*** pc_m has joined #openstack-infra11:04
*** afazekas has quit IRC11:05
*** yfried has joined #openstack-infra11:06
*** yfried__ has quit IRC11:07
*** e0ne is now known as e0ne_11:07
*** e0ne_ is now known as e0ne11:07
*** pc_m has quit IRC11:09
*** pc_m_ has joined #openstack-infra11:09
*** sdake has quit IRC11:10
*** marcusvrn has joined #openstack-infra11:11
*** dprince has joined #openstack-infra11:11
*** marcusvrn1 has joined #openstack-infra11:12
*** yamada-h has quit IRC11:12
*** jamespage_ has joined #openstack-infra11:13
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Log heat-coe-templates to the openstack-containers channel
*** marcusvrn has quit IRC11:15
*** pc_m_ has quit IRC11:18
*** e0ne is now known as e0ne_11:19
*** pc_m has joined #openstack-infra11:19
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
*** afazekas has joined #openstack-infra11:21
*** cdent has joined #openstack-infra11:23
*** salv-orlando has quit IRC11:23
*** dhritishikhar has joined #openstack-infra11:24
*** otter768 has joined #openstack-infra11:24
*** dkranz has quit IRC11:25
*** hdd_ has joined #openstack-infra11:25
deepakcsCan someone help me understand the significance of "if [ "$BRANCH_OVERRIDE" != "default" ] ; then ..." in jenkins/jobs/devstack-gate.yaml ?11:25
*** dkranz has joined #openstack-infra11:25
*** Mithrandir_ is now known as Mithrandir11:25
*** hdd has quit IRC11:27
*** coolsvap is now known as coolsvap|afk11:27
*** e0ne_ has quit IRC11:29
*** otter768 has quit IRC11:29
*** markvoelker has joined #openstack-infra11:29
rcarrillocruzdeepakcs: i guess it's because you can run different jobs per branch?11:29
rcarrillocruzdifferent logic depending if it's default or not11:29
*** mpaolino has quit IRC11:30
deepakcsrcarrillocruz, branch of what ? not clear to me11:30
*** jamespage_ has quit IRC11:31
*** jamespage__ has joined #openstack-infra11:32
*** zul has joined #openstack-infra11:32
rcarrillocruzof the project being tested11:32
rcarrillocruzin this case, devstack-gate11:32
rcarrillocruzcheck it out projects.yaml:11:32
AJaegerdeepakcs, jenkins/jobs/projects.yaml - line 200111:32
*** pc_m has quit IRC11:32
AJaeger"branch-override: stable/icehouse"11:33
rcarrillocruzyou pass branch-override per project11:33
deepakcsAJaeger, ah ok, so its the branch of project (cinder, glance etc) to test11:33
*** markvoelker has quit IRC11:33
*** pc_m has joined #openstack-infra11:34
deepakcsAJaeger, so if my job-template in devstack-gate.yaml does not use "if [ "$BRANCH_OVERRIDE" != "default" ] ..." - what does it mean ? Does it mean i cannot test anything but latest/master of the projects ?11:34
*** ddieterly has joined #openstack-infra11:35
*** pc_m has quit IRC11:35
deepakcsAJaeger, rcarrillocruz for eg: devstack-gate.yaml line: 1911 - devstack plugin template does not use branch-override at all11:35
AJaegerdeepakcs, let's better ask sdague and fungi...11:35
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Add cloud-config-hosts element for non-puppet jobs
AJaegerbranch-designator allows also to have different conditions (voting/non-voting) for one job11:36
*** pblaho has joined #openstack-infra11:37
*** markvoelker has joined #openstack-infra11:38
*** ddieterly has quit IRC11:39
*** pc_m has joined #openstack-infra11:41
deepakcsAJaeger, ya, i know about branch-designator, asking about the branch-override not being used in the devstack plugin job template11:43
*** Ala has quit IRC11:43
AJaegerdeepakcs, those are often used together - but yeah, I also have some questions here ;)11:44
deepakcsAJaeger, ok :) i wanted to understand the relation between designator and override if any.. bcos i am seeing soem strange failure in one of the patch (it says duplicate job check-tempest-dsvm-glusterfs) exists11:44
*** ldnunes has joined #openstack-infra11:44
*** salv-orlando has joined #openstack-infra11:45
*** mrmartin has joined #openstack-infra11:45
*** dkranz has quit IRC11:47
AJaegerdeepakcs, duplicate via projects.yaml, line 3549 on project devstack-plugin-glusterfs: '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}'11:47
*** alexpilotti has joined #openstack-infra11:47
deepakcsAJaeger, inspite of no duplicate jobs being present there ( I have 2 jobs -nv and one without -nv) but it seems its seeing both of them as same, hence the dup11:47
AJaeger'{pipeline}-tempest-dsvm-full-glusterfs{branch-designator}' and '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' gives the duplicate11:48
deepakcsAJaeger, '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' and '{pipeline}-tempest-dsvm-full-glusterfs{branch-designator}' are _not_ duplicates, right ? from a job template perspective ?11:48
AJaegerwith plugin=glusterfs11:48
deepakcsAJaeger, why ?11:48
*** mwagner_lap has quit IRC11:48
deepakcsAJaeger,  but there is no job with plugin: glusterfs11:48
*** pc_m has quit IRC11:48
AJaegerdeepakcs, they are duplicates, both the same name, aren't they?11:48
*** chmouel has quit IRC11:48
*** tnovacik has quit IRC11:48
deepakcsAJaeger, but the name matchign happens before the yaml params are expanded, right ?11:48
*** pc_m has joined #openstack-infra11:48
deepakcsAJaeger, both are not same, one has {plugin} and other has glusterfs in the same place11:49
openstackgerritMerged openstack-infra/project-config: Log heat-coe-templates to the openstack-containers channel
AJaegerdeepakcs, the name matchin is after expansion11:49
*** dkranz has joined #openstack-infra11:49
AJaegerYou have '{pipeline}-tempest-dsvm-full-glusterfs{branch-designator} which is check-tempest-dsvm-full-glusterfs"11:49
AJaegerand you have '{pipeline}-tempest-dsvm-full-{plugin}{branch-designator}' which gives the same name11:50
*** chmouel has joined #openstack-infra11:50
deepakcsAJaeger, if so, then there is no job with plugin: glusterfs in the yaml file (the patch replaces plugin: glusterfs as we are creating a specific glusterfs job)11:50
AJaegerdeepakcs, scroll back: "line 3549 in projects.yaml"11:50
deepakcsAJaeger, thats the line the patch under Q changes :)11:51
deepakcsAJaeger, and remove plugin: glusterfs11:51
AJaegerlet me double check your patch...11:51
deepakcsAJaeger, sure, thanks11:51
*** baoli has joined #openstack-infra11:51
deepakcsAJaeger, also a related Q... if the name matching happens _after_ expansion, how does it know what to match, bcos job-templates in devstack-gate.yaml has parameterized names (not with expanded ones)11:51
deepakcsAJaeger, are u sure that the name matchin is after expansion ?11:52
openstackgerritMerged openstack-infra/project-config: Adding create permissions to networking-l2gw
openstackgerritMerged openstack-infra/project-config: Add Ironic python3.4 experimental job
*** fhubik_afk is now known as fhubik11:52
openstackgerritMerged openstack-infra/project-config: Add Elastic Recheck to
*** fhubik is now known as fhubik_afk11:53
AJaegerdeepakcs, just rename your job ;)11:53
deepakcsAJaeger, some black magic happenign there ?11:54
AJaegerAFAIK you can say both: "gate-{name}-pep8" with "name: bla" and "gate-bla-pep8"11:54
*** afazekas has quit IRC11:54
deepakcsAJaeger, you couldn't figure why its giving dup ?11:54
AJaegerjjb will expand it - and the same is here, jjb could expand your job in two different ways11:54
*** dkranz has quit IRC11:55
deepakcsAJaeger, jjb should do that only if plugin: glusterfs was defined for some job,but that isn't the case, since the patch remove that line11:55
deepakcsAJaeger, so still unclear on how/why it sees 2 dup jobs11:55
openstackgerritMerged openstack-infra/project-config: Check more Puppet modules with tripleo-f20puppet-nonha
*** dkranz has joined #openstack-infra11:56
AJaegerdeepakcs, devstack-gate.yaml still has two jobs that can be used for your line.11:56
*** chmouel_ has joined #openstack-infra11:57
deepakcsAJaeger, Hmm interesting11:57
AJaegerdeepakcs, I suggest to discuss with clarkb or fungi, they might be able to give you a better explanation.11:57
*** chmouel has quit IRC11:57
deepakcsAJaeger, ok one quick observation. I removed line 2044 from jenkins/jobs/devstack-gate.yaml and then it no longer gives dup job error. Any idea how removing that makes it work ?11:57
*** zhiwei has joined #openstack-infra11:58
*** sileht has quit IRC11:59
AJaegerdeepakcs, interesting ;) no quick idea11:59
*** e0ne has joined #openstack-infra11:59
AJaegerPlease do rename your job, we know it will cause problems, so let's name them differently. And if the patch removes the usage of the old template, let's do it properly and remove it everywhere.12:00
*** Longgeek has quit IRC12:00
*** ociuhandu has joined #openstack-infra12:00
*** ociuhandu has quit IRC12:00
AJaegerAnd in that case I wonder why you cannot just change the old template?12:01
*** alexpilotti has quit IRC12:01
*** dizquierdo has joined #openstack-infra12:01
deepakcsAJaeger, change old template meaning ?12:01
deepakcsAJaeger, old template was the generic devstack plugin template12:01
*** woodster_ has joined #openstack-infra12:01
AJaegerdeepakcs, ah, you're right - that shouldn't be touched12:01
deepakcsAJaeger, we tried to add tempest specific regex in that old template for 'glusterfs' and sdague didn't like it.. he suggested to add new job, hence this new patch :)12:02
deepakcsAJaeger,  i hopeu remember giving +2 and then removign it after sdague comment :)12:02
AJaegerdeepakcs, I have to leave now for a meeting... but yes, there was something...12:02
deepakcsAJaeger, ok thanks for ur time12:02
*** yfried_ has joined #openstack-infra12:03
*** liam__ has joined #openstack-infra12:04
*** Ala has joined #openstack-infra12:05
*** Longgeek has joined #openstack-infra12:05
*** openstackgerrit has quit IRC12:06
*** yfried has quit IRC12:06
*** openstackgerrit has joined #openstack-infra12:06
*** pc_m has quit IRC12:07
*** afazekas has joined #openstack-infra12:07
*** sileht has joined #openstack-infra12:09
*** dkranz has quit IRC12:09
*** dkranz has joined #openstack-infra12:11
deepakcsAJaeger, fwiw, we found the root cause, adding a job entry under devstack-jobs (  line 2044 from jenkins/jobs/devstack-gate.yaml ) causes check-tempest-dsvm-full-glusterfs job to be defined and we are also defining it under our job definition, hence the dup12:12
*** bswartz has quit IRC12:12
*** u_glide has left #openstack-infra12:12
deepakcsAJaeger, the correct way to fix it is to remove line 2044 from jenkins/jobs/devstack-gate.yaml - its not needed there, we create our jobs so no need to add our job-template under devstack-jobs section12:12
*** weshay has joined #openstack-infra12:13
mordredsaggi: morning12:14
saggimordred: morning12:14
*** deepakcs has quit IRC12:15
*** esker has quit IRC12:15
AJaegerdeepakcs, I don't think that removing is the proper fix, please rename12:15
mordredin reverse order - one of us can change your github description - but it's not a thing we want to update a lot :)12:16
mordredsaggi: also - as for ff-merge - let me look at your patch12:16
openstackgerritJulien Danjou proposed openstack-dev/hacking: tests: allow to run the same test for different code
saggimordred: Thanks, We don't intend to change it a lot :).12:17
*** mpaolino has joined #openstack-infra12:18
*** mpaolino has quit IRC12:18
mordredsdague: cool. just let me know what to change it to12:18
AJaegerfungi, mordred : I'm not seeing the latest merges to project-config in the post queue using
mordredAJaeger: yay!12:18
AJaegerany idea what's broken? OTher jobs go in..12:19
*** dprince has quit IRC12:19
*** fhubik_afk is now known as fhubik12:20
mordredAJaeger: I see things in the post queue?12:20
mordredsaggi: cool. just let me know what to change it to12:20
saggimordred: Whatever it is now. We already updated the project-config12:21
openstackgerritBharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed
openstackbug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] - Assigned to Bharat Kumar Kobagana (bharat-kobagana)12:21
mordredsaggi: also - as for your change to use FF-only - I believe that change to your project-config will work - but I'll warn you it's gonna be super annoying to have gerrit set up that way12:21
mordredsaggi: gotcha12:21
mordredsaggi: what are you trying to accomplish with ff-only?12:22
*** BobBall_ is now known as BobBall12:22
*** dkranz has quit IRC12:22
*** fhubik is now known as fhubik_afk12:25
*** liam__ has quit IRC12:25
openstackgerritMerged openstack-infra/project-config: New Fuel plugin project for Calico
saggimordred: As I said in my commit message, Gerrit+Merge commits are a bad idea. They break most of gits helpful features. We currently don't have a lot of tests and do a lot of infra changes so we want to make sure developers test against master before we commit. We will probably change to cherry-pick later. If this is an issue we could just move to cherry-pick instead.12:25
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Add cloud-init-hosts element for non-puppet jobs
*** fhubik_afk is now known as fhubik12:27
saggimordred: I honestly don't understand why I am the first to bring this up. Doesn't anybody care that the git history is a mess. But I'm not here to start a crusade, I just want my master branch to be readable.12:27
*** Hal has joined #openstack-infra12:29
*** Hal is now known as Guest2037412:29
openstackgerritMerged openstack-infra/project-config: Support feature/development for askbot-theme
*** jamespage__ has quit IRC12:30
*** gordc has joined #openstack-infra12:30
*** bknudson has quit IRC12:31
*** claudiub has joined #openstack-infra12:31
*** chmouel_ is now known as chmouel12:33
mordredsaggi: I hear that - it's just an unworkable option on most of our projects due to velocity12:35
*** rfolco has joined #openstack-infra12:35
saggimordred: Even cherry-pick?12:36
mordredsaggi: also, cherry-pick in gerrit gets very confusing for end users12:36
*** dprince has joined #openstack-infra12:36
*** samuelBartel has joined #openstack-infra12:36
*** ddieterly has joined #openstack-infra12:36
*** kgiusti has joined #openstack-infra12:36
mordredit doesn't _quite_ work the way people think it will - we had it turned on for sevearl of the infra projects for a while12:36
*** Longgeek has quit IRC12:36
mordredit was more of a headache than it was worth12:36
mordredI'd try ff-only if you feel strongly about not seeing merge commits and see how it works for you12:37
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Using Neutron network by default in gate-rally-dsvm-murano-rally
*** openstackgerrit has quit IRC12:37
mordredhowever - you'll find that 100% of our infrastructure around here is built on the assumption that we're running merge commits12:37
*** openstackgerrit has joined #openstack-infra12:37
mordredso it's entirely possible that things will break in unexpected ways12:37
saggimordred: I worked with gerrit+cherry-pick for years. Had no issues.12:38
saggimordred: hmmmm12:38
*** Longgeek has joined #openstack-infra12:39
*** ddieterly has quit IRC12:40
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Using Neutron network in gate-rally-dsvm-murano-rally
mordredsaggi: we have approx 2k developers who have very basic git knowledge - most of them perform the one or two commands we tell them to perform - so "confusing for end users" for us is always trying to look through that lens. _we_ can handle the ramifications of gerrit cherry-pick - but when running it it required enough git smarts on the user side that it seemed pretty unworkable for our general12:41
mordredthere are a few places where we explicitly don't do "nicer" things for that reason - and it can be annoying for folks with deeper git-fu12:42
openstackgerritLingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta
*** ayoung has joined #openstack-infra12:43
*** xylan has joined #openstack-infra12:43
*** rossella_s has quit IRC12:44
*** rossella_ has quit IRC12:44
*** rossella_s has joined #openstack-infra12:44
*** miqui has joined #openstack-infra12:45
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade
*** BharatK has quit IRC12:47
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Add acl rule for Mistral to allow creating branches
xylanhi there, i proposed a new stackforge project,, but i didn't know whether or not everything is ok, i just did this according to
*** fhubik is now known as fhubik_afk12:48
*** bswartz has joined #openstack-infra12:49
*** abregman has quit IRC12:49
*** jamespage_ has joined #openstack-infra12:50
saggimordred: I understand, I do like my git history. It's so useful. But I don't want to go against the grain if that means it might break something in the workflow.12:51
*** sileht has quit IRC12:52
AJaegermordred, there are things in the post queue but project-config is not in it at all ;(12:55
mordredAJaeger: oh! project-config patches are not doing post queue12:55
AJaegermordred, indeed12:56
*** bknudson has joined #openstack-infra12:56
mordredAJaeger: I blame jaypipes, but I have no reason to do so12:56
* jaypipes hides12:57
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral
AJaegerxylan, you did not setup irc access properly, see
xylanAJaeger: thanks for feedback...I see that. but, how could i create a channel?13:00
AJaegerxylan, you setup #terracota in your change - if you don't want it, then don't add it ;)13:01
*** jamesmcarthur has joined #openstack-infra13:02
xylanAJaeger: i really need it. but i didn't know how to create it13:02
*** jamespage_ has quit IRC13:03
AJaegerxylan, check the URL for comments on how to setup a new channel - and read the freenode docs13:03
*** jamespage_ has joined #openstack-infra13:03
xylanAJaeger: ok, thanks very much!13:03
AJaegerxylan, patch looks fine otherwise13:03
*** fhubik_afk is now known as fhubik13:03
xylanAJaeger: :)13:04
*** jamespage_ has quit IRC13:04
* morganfainberg waves at jeblair.13:05
fungiAJaeger: which change merging should have triggered which post jobs for project-config?13:05
AJaeger is the one that triggered my research13:05
AJaegerBut then checking status.o.o/zuul I say not a single job in the post queue for project-config - and none entering it.13:06
AJaegerWhile other projects entered easily13:06
AJaegerso, is still not created13:07
fungiAJaeger: i believe the only current post jobs for it are scoped to changes for specific file patterns. unless changes touch files in the docs-site or specs directories or the tox.ini file, there's nothing to be triggered13:07
*** jtriley has joined #openstack-infra13:07
AJaegerah, you're right - post jobs is only specs site and docs-site.13:08
fungiAJaeger: see the files scoping for ^(check-infra-docs-index|publish-infra-docs-index)$ and ^(check-generate-specs-site|publish-specs-site)$ in the jobs list of layout.yaml13:08
AJaegerBut docs-site did not trigger - or did not work as expected13:08
AJaegerfungi, yeah, just double checked13:08
fungiyeah, that job might also be broken13:08
*** ZZelle has joined #openstack-infra13:08
AJaeger177077 should have triggered. The check-infra-docs-index was run13:09
* jeblair waves at morganfainberg13:09
*** alexpilotti has joined #openstack-infra13:10
AJaegerso, why did publish-specs-site not trigger? Is there a problem running this in post?13:10
* mordred waves at morganfainberg and jeblair13:10
fungiAJaeger: i'm looking for logs now13:10
morganfainbergmordred: ohai!13:10
fungihrm, gives me a 40413:10
AJaegerI meant: publish-infra-docs-index. Thanks, fugni for looking13:10
AJaegerfungi, or should it be: ? Also 40413:11
fungioh, good point. i needed to dig out the merge commit sha13:11
AJaegermerge commit sha is the one above13:12
AJaegerif it's really the merge commit that'S the key: will checking for files really work in the post queue?13:13
AJaegerThe match for 'docs-site/.*' will not work for the merge sha, or does it?13:13
fungithat's a good point too. the specs site used it, but...13:13
*** dustins has joined #openstack-infra13:14
fungiit's entirely possible we've also never used that job either since that got restricted13:15
AJaegermight be13:15
mordredsdague: hey - so, you use devstack-vagrant, yeah? can I use it to get a single-node devstack?13:16
sdaguemordred: yes13:16
sdaguethere is a config.yaml (sample included)13:16
sdagueif you only define hostname_manager, and not hostname_compute, you get a single node13:16
fungiAJaeger: looks like you restricted that last month with
fungiso we might just need to take that file restriction off for the publish versions13:17
*** dims has quit IRC13:17
mordredsdague: cool13:17
sdagueit should probably be documented better, but it's a pretty spare time thing, so I haven't gotten around to it13:17
* AJaeger hides13:17
fungiAJaeger: and let it run for every change post merge13:17
AJaegerfungi, ok, will change both jobs13:17
*** dims has joined #openstack-infra13:18
mordredsdague: also - and I'm about to show my extreme ignorance here13:18
mordredsdague: I just got an error in trying to stand up a normal vm that something about not being able to run kvm and virtualbox13:18
fungiAJaeger: now that you mention it, i do recall files sections only being valid for proper change pipelines (i.e. those which act on gerrit change ids)13:18
mordredsdague: do I just rmmod kvm to make kvm go away right now?13:18
sdaguehmmmm.... that seems weird, I think I have both running on my boxes13:18
fungiAJaeger: so that is almost certainly the problem13:18
sdagueI suppose, but I didn't think I'd run into that issue before13:19
AJaegerfungi, thanks! Learned something new...13:19
mordredsdague: VT-x is being used by another hypervisor13:19
mordredsdague: virtualbox can't operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot.13:20
sdaguehmm... weird13:20
mordredsdague: :)13:20
sdaguebecause I don't get that error :)13:20
mordredsdague: have I mentioned that I've never gotten vagrant to boot a machine ever13:20
sdaguekernel 3.19.013:20
sdaguemordred: you have not mentioned that fact13:20
mordredsdague: I'm going to make it happen today13:20
sdaguevagrant                                              1:1.6.313:20
AJaegerour gerritbot is dead, can anybody restart it?13:21
*** Ala has quit IRC13:21
fungiAJaeger: i'll whip up an addition to to that effect too13:21
AJaegerfungi, is the change I just pushed13:21
fungiAJaeger: restarting it now13:21
mordredsdague: 1.6.5+dfsg1-213:21
*** yamahata has quit IRC13:21
*** tiswanso has joined #openstack-infra13:21
sdagueI'm trying to remember if there was anything I needed to once tweek in virtual box13:21
mordredsdague: with 4.3.26-dfsg-2ubuntu1 of virtualbox13:21
AJaegerfungi, let's get that updated content to ;)13:21
*** openstackgerrit has quit IRC13:21
*** openstackgerrit has joined #openstack-infra13:22
sdaguevirtualbox-4.3                                       4.3.26-98988~Ubuntu~raring13:22
sdagueso... I'm running virtualbox from oracle packages13:22
*** esker has joined #openstack-infra13:22
fungiAJaeger: okay, openstackgerrit has been restarted. hopefully its silence is not a sign that gerrit stream-events is dead again13:22
AJaegerfungi, should we add a comment to the beginning of the jobs section in zuul/layout.yaml13:22
sdagueI do no know if that's important to this or not13:23
AJaeger"Queue lengths: 5282 events, 77 results.  " in zuul13:23
*** rkukura has quit IRC13:23
*** russellb has quit IRC13:24
fungilast stream event seen by zuul was 13:09 utc (15 minutes ago)13:24
fungihopefully that's just the queue length13:24
sdaguemordred: I am happy to compare config values if you find things aren't working, I don't think I've had an issue recently getting vagrant / vbox up on ubunut13:24
AJaegermy project-config change is not in the queue ;(13:25
*** jtriley_ has joined #openstack-infra13:25
AJaegerfungi, seems we have a problem again13:25
*** otter768 has joined #openstack-infra13:25
jeblairAJaeger: wait until zuul's queue is empty to make that determination13:25
AJaeger"Queue lengths: 0 events, 0 results."13:25
jeblairAJaeger: okay then :)  my wifi is slow13:26
fungiunfortunately, i'm not on a fast enough computer to load the zuul status page yet13:26
jeblairAJaeger, fungi: yeah, looks stuck13:26
*** pc_m has joined #openstack-infra13:26
*** 7JTAAIHSL has quit IRC13:26
jeblair01861a4b waiting .... 13:10:07.270
jeblairlots of those13:27
jeblairi guess my patch still isn't deployed :/13:27
*** tnovacik has joined #openstack-infra13:27
*** eharney has quit IRC13:27
fungii'll get a good set of thread dumps and then restart13:27
TheJuliaw/win 913:27
*** russellb has joined #openstack-infra13:28
*** jtriley has quit IRC13:28
*** peristeri has joined #openstack-infra13:28
*** Ala has joined #openstack-infra13:28
*** mriedem has joined #openstack-infra13:29
*** otter768 has quit IRC13:29
jeblairfungi: when you're done, what's the status of the gerrit build with the debugging patch?13:29
*** marcusvrn1 has quit IRC13:30
*** superdan is now known as dansmith13:31
*** devvesa has joined #openstack-infra13:32
fungijeblair: zaro was unable to reproduce (or even get gerrit running normally) with the version of gerrit we're running now, but the patch seems to be working when picked onto 2.10 and since it's not long until the kilo release and we're nearing the gerrit upgrade i was reluctant to schedule a lengthy downtime for gerrit (since testing that again will result in another lucene reindex)13:32
*** sileht has joined #openstack-infra13:32
*** mestery_ is now known as mestery13:32
fungii've got jstack thread dumps in my homedir, named jstack.1 through 313:32
fungiand gerrit is restarted again13:32
*** BobH has joined #openstack-infra13:34
*** arielb has joined #openstack-infra13:34
*** Ala has quit IRC13:34
*** ddieterly has joined #openstack-infra13:34
*** dboik has quit IRC13:34
*** Ala has joined #openstack-infra13:34
jeblairyeah, but if this is going to keep happening every few hours for the next two weeks, i'd love it if we could make progress on debugging it13:35
openstackgerritLingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta
AJaegerthanks, fungi!13:35
mordredsdague: ok - we're one step closer to having achieved this tool working for me13:35
mordredsdague: I now have an instance booted, but ssh keys don't seem to be set up?13:35
fungiagreed. i've not seen it happen this close together before today13:35
mordredsdague: mordred@camelot:~/vms$ ssh -p 2222 vagrant@localhost13:35
mordredvagrant@localhost's password:13:35
*** jamesmcarthur has quit IRC13:35
jeblairfungi: just so we're clear -- i _ran_ the patch locally13:36
mordredsdague: (this isn't using devstack-vagrant yet - I did a 'vagrant init ubuntu/trusty64 ; vagrant up' - just to make sure I could get that far13:36
jeblairthere is no reason i can think of that there is something wrong with the patch.  we have an undiagnosed problem with the build13:36
sdaguemordred: does "vagrant ssh" work ?13:36
mordredsdague: AH!13:36
*** marcusvrn1 has joined #openstack-infra13:36
mordrednever would have thought of that13:36
mordredsdague: yes13:37
mordredit does13:37
sdagueyeh, so vagrant ssh is for built in connection to all gutets13:37
sdagueconfig.yaml supports adding stack_sshkey13:37
sdagueso you can13:37
fungijeblair: do you have a different build which works? i just also don't want to have gerrit offline for hours while we try to figure out what about that build is broken. but you're right if it's going to start continually breaking now then better that we make progress while it's offline than have it not triggering ci jobs13:37
mordredwell, I'll try devstack-vagrant now13:37
sdaguessh stack@....13:37
AJaegerfungi, do we need to send a note out to ask people to requeue jobs - or will you do it automatically somehow?13:37
jeblairfungi: yeah, i think we should download gerrit-v2. and run it locally on some machine to make sure the problem is reproduced, then create another build and run it locally to make sure it does not appear, then upload that manually to tarballs and deploy it13:39
fungijeblair: zaro says he downloaded that and it ran fine13:39
fungifor him13:39
jeblairfungi: i'll see if i can find the build i used for testing13:39
fungiAJaeger: yes, i'll switch to another room and work out the details13:40
jeblairfungi: oh, i thought he never got it working.  i must have bisunderstood13:40
*** Guest20374 has quit IRC13:40
AJaegerfungi, thanks! Are you travelling?13:40
mordredsdague: do I put the filename of the ssh key? or the contents?13:40
fungijeblair: he was never able to get any build of gerrit to start with contactstore enabled13:40
*** Hal has joined #openstack-infra13:40
fungifor reasons he's unable to identify yet13:40
sdaguemordred: the pub key data13:40
*** Hal is now known as Guest8909413:40
sdagueso, contents13:40
fungiAJaeger: no, i just made the mistake of looking at irc from bed when i first woke up, and have been trapped there ever since13:40
sdagueoh, hmm.... that's not in the sample documentaiton13:41
fungiwaiting on a lull in the crazy13:41
AJaegerfungi, take a break, drink a coffee ;)13:41
sdagueI should fix that13:41
sdagueoh, no, it's there, it's just not as clearly docced as it could be13:42
*** Guest89094 has quit IRC13:42
openstackgerritMerged openstack-dev/pbr: Honour new test variable PIPVERSION
*** pc_m has quit IRC13:43
openstackgerritMatthew Treinish proposed openstack/requirements: Add os-testr to global requirements list
mtreinishsdague: if you get a sec just adds the tempest-lib kilo cap13:44
sdaguemtreinish: +213:44
*** morganfainberg has quit IRC13:44
*** jroll has quit IRC13:44
*** jroll has joined #openstack-infra13:44
AJaegersdague, mordred could you review , please?13:45
mtreinishsdague: cool, thanks13:45
*** xianghui has quit IRC13:45
fungi#status notice gerrit has been restarted to clear a problem with its event stream. change events between 13:09 and 13:36 utc should be rechecked or have approval votes reapplied as needed to trigger jobs13:46
openstackstatusfungi: sending notice13:46
*** fhubik is now known as fhubik_afk13:46
*** sergsh has quit IRC13:46
*** xianghui has joined #openstack-infra13:46
-openstackstatus- NOTICE: gerrit has been restarted to clear a problem with its event stream. change events between 13:09 and 13:36 utc should be rechecked or have approval votes reapplied as needed to trigger jobs13:47
*** morganfainberg has joined #openstack-infra13:47
AJaegerthanks, sdague, fungi13:47
anteayajhesketh: when you say you can't do a query with an offset, what query were you trying to run? you probably are correct, I'm just curious13:47
openstackstatusfungi: finished sending notice13:48
*** bookwar has joined #openstack-infra13:49
*** fhubik_afk is now known as fhubik13:51
*** xylan is now known as xylan_13:51
*** xylan_ is now known as xylan13:52
*** Ala has quit IRC13:54
openstackgerritMerged openstack-infra/project-config: Fix project-config post jobs
*** mkoderer has quit IRC13:55
*** dboik has joined #openstack-infra13:55
*** vishy has quit IRC13:55
openstackgerritMonty Taylor proposed openstack-infra/shade: Add inventory command to shade
*** mwagner_lap has joined #openstack-infra13:57
*** amitgandhinz has joined #openstack-infra13:57
*** vishy has joined #openstack-infra13:58
*** mpaolino has joined #openstack-infra13:58
*** eharney has joined #openstack-infra13:58
*** mkoderer has joined #openstack-infra13:58
*** funzo has joined #openstack-infra13:59
*** rossella_s has quit IRC13:59
*** liam__ has joined #openstack-infra14:01
*** tonytan4ever has joined #openstack-infra14:01
*** ibiris is now known as ibiris_away14:01
*** liam__ has quit IRC14:02
*** alexpilotti has quit IRC14:02
openstackgerritJulia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through
*** ibiris_away is now known as ibiris14:03
*** wznoinsk has joined #openstack-infra14:03
*** sergsh has joined #openstack-infra14:04
*** sigmavirus24_awa is now known as sigmavirus2414:05
yolandahi, i need some clarification for usage of zookeeper on the tests, is that still actively used?14:05
*** hodos|2 has joined #openstack-infra14:05
*** dkranz has joined #openstack-infra14:06
fungii've been working on the assumption that it is14:06
fungifor tooz at least?14:07
yolandafungi, i wasn't aware of that, so i wanted to know more, as i had some problems with that14:07
mordredyeah. I mean, it's not used for most of the things, right?14:07
*** samuelBartel has quit IRC14:07
mordredtooz and maybe ceilometer uses tooz at this point for something?14:07
*** xylan has left #openstack-infra14:07
fungiyolanda: problems installing the packages, or problems using zookeeper?14:07
yolandawell, zookeeper not booting, looked as a permissions problem14:07
yolandabut i got surprised because i had no idea that we had zookeeper on the slaves14:08
yolandai fixed the issue but was a surprising discover14:08
*** fhubik is now known as fhubik_afk14:08
*** marcusvrn1 has quit IRC14:09
*** stevemar has joined #openstack-infra14:10
*** dkliban has joined #openstack-infra14:10
*** rkukura has joined #openstack-infra14:11
*** btully has joined #openstack-infra14:11
*** fhubik_afk is now known as fhubik14:13
*** zhiwei has quit IRC14:13
*** dmellado has joined #openstack-infra14:14
jheskethanteaya: against the gerrit api, it seems that the --start param was introduced after 2.814:15
jhesketh(it's in the head documentation, not sure if that means 2.9 or 2.10)14:15
anteayajhesketh: ah okay thanks14:15
*** jtriley_ has quit IRC14:16
*** jtriley has joined #openstack-infra14:17
*** mtanino has joined #openstack-infra14:17
anteayajhesketh: I can't find docs for the --start param anywhere so far14:18
nibalizeryolanda: mordred fungi so maybe 'apt-get install' zookeeper could move out of the slave definition and into the test that needs it?14:19
yolandaif it's only used in a small subset of patches, it should be goog removing from slaves14:19
yolandawe have that on some static jenkins slaves, it's a process running there forever, that may not be even used14:20
mordrednibalizer: yup14:20
mordrednibalizer: we have a general desire to do that for everything14:20
nibalizeryea so lets do that14:21
mordrednibalizer: so, we'd like to have very few things in the base slave definition, one slave type with caching, and install appropriate things in the tests14:21
nibalizeryea, swinging of the pendulum etc etc14:21
nibalizeri like it14:21
nibalizerbig tent needs a really pared down node14:21
funginibalizer: i'm working on that for _all_ packages14:21
funginot just zookeeper14:21
nibalizerfungi: ++14:21
fungiso yes, zookeeper can be scoped back to just specific jobs soon, along with a lot of other packages too14:22
*** salv-orl_ has joined #openstack-infra14:22
openstackgerritKyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot
*** salv-orlando has quit IRC14:23
*** mase_x200 has joined #openstack-infra14:24
*** tiswanso has quit IRC14:24
*** marcusvrn has joined #openstack-infra14:25
*** ChuckC_ has joined #openstack-infra14:25
*** ChuckC has quit IRC14:26
*** wenlock has joined #openstack-infra14:26
*** wenlock_ has joined #openstack-infra14:26
AJaegerjaypipes, you can unhide again from mordred - it was my own error that caused the problem I run into ;(14:28
openstackgerritMerged openstack-infra/project-config: Add acl rule for Mistral to allow creating branches
jaypipesAJaeger: :)14:28
* jaypipes crawls out from hole14:28
*** michchap has quit IRC14:28
AJaegerSorry, jaypipes ! Hope you didn't hurt...14:28
*** mattfarina has joined #openstack-infra14:29
* mordred hands jaypipes a towel14:30
*** ChuckC_ has quit IRC14:30
jeblairnibalizer, yolanda: it's a constituent of the dib-nodepool infra priority effort14:30
gordcdtroyer: just curious, but do you know of any clients outputting json rather than typical print_dict/list formatting?14:30
*** sputnik13 has joined #openstack-infra14:30
mordredgordc: like, command line clients?14:30
gordcdtroyer: i see this but it seems like it was abandoned:
openstackLaunchpad bug 1254351 in python-novaclient "client should have an option to output json" [Undecided,Won't fix] - Assigned to Abhishek Chanda (abhishek-i)14:30
yolandajeblair, is that the work that is driven by fungi?14:30
gordcmordred: yeah14:30
mordredgordc: python-openstackclient has an option for json output14:31
mordredgordc: occasionally glanceclient outputs json - but when it does, it's a bug :)14:31
*** dhritishikhar has quit IRC14:31
*** dhritishikhar_ has joined #openstack-infra14:31
gordcmordred: :) bugs are hidden features.14:31
gordcmordred: thanks for pointer. i'll take a look.14:32
fungiyolanda: yes14:32
*** Ala has joined #openstack-infra14:32
anteayamordred: a towel, that might the first useful thing I've ever seen you hand to anyone14:32
yolandafungi, i need to take a deeper look at your job, is one of the parts that is black magic to me14:32
*** dhritishikhar_ has quit IRC14:32
mordredanteaya: well, he _was_ in a hole14:33
anteayahe was14:33
*** dhritishikhar_ has joined #openstack-infra14:33
*** achanda has joined #openstack-infra14:33
openstackgerritVictor Stinner proposed openstack-infra/project-config: Make oslo.messaging py3 job voting
*** tiswanso has joined #openstack-infra14:34
fungiyolanda: is a preview/proof of concept. i'm presently working on making bindep solve that "base-packages-trusty" builder in a more cross-platform friendly manner and reusing it to also determine what to cache on our worker images14:35
*** weshay has quit IRC14:35
*** Longgeek has quit IRC14:35
fungiif things calm down today, i've got a bunch of held workers in rackspace and hpcloud representative of all our devstack-.* and corresponding bare-.* types for some final bindep testing before i tag its current state up to pypi14:35
*** Longgeek has joined #openstack-infra14:36
mordredfungi: woot14:36
* yolanda needs to dig more about bindep14:37
*** tjones1 has joined #openstack-infra14:37
fungionce that part works, the package installer macro can look for a "other-requirements.txt" in the top-level directory of a project's git tree, and if found use that otherwise fall back to using
*** dangers_away is now known as dangers14:38
*** weshay has joined #openstack-infra14:39
fungiand we can tackle some of the less common entries in that fallback list piecemeal, for example getting a usable other-requirements.txt into ceilometer and tooz, and then removing entries from the fallback14:39
*** jtriley has quit IRC14:40
*** achanda has quit IRC14:41
*** dmellado has quit IRC14:42
*** jamesmcarthur has joined #openstack-infra14:42
*** saggi has quit IRC14:43
openstackgerritVictor Stinner proposed openstack-infra/project-config: Make oslo.messaging py3 job voting
*** spzala has joined #openstack-infra14:48
*** zz_jgrimm is now known as jgrimm14:48
*** hemnafk is now known as hemna14:49
*** jtriley has joined #openstack-infra14:50
sdaguegrr... pypi mirrors in rax randomly dropping connections14:50
*** annegentle has joined #openstack-infra14:51
mordredsdague: it's a 'feature'14:51
mordredsdague: zomg. vagrant is actually doing things on my box without dying14:52
mordredanteaya: ^^ you'll be amazed14:52
anteayamordred: I am amazed14:52
mordredsdague: seriously. this is the first time it has ever done anything other than a crazy cryptic error14:52
anteayaI never thought I would see the day14:52
*** tkelsey has joined #openstack-infra14:52
*** ibiris is now known as ibiris_away14:53
openstackgerritLingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta
anteayaI'm impressed you were able to work through your rage and get it working14:53
mordredsdague: also - if you clean up docs on the ssh key parameter - it's worth noting that it's just the key contents, not the leading ssh-rsa or the trailing comment14:53
*** samuelBartel has joined #openstack-infra14:53
*** salv-orl_ has quit IRC14:54
sdagueoh, yeh. Good point14:55
*** jtriley_ has joined #openstack-infra14:56
*** ibiris_away is now known as ibiris14:56
greghaynesmordred: in config-drive?14:57
*** _nadya_ has quit IRC14:58
*** ildikov has quit IRC14:58
*** jtriley has quit IRC14:59
*** zul has quit IRC14:59
*** marcusvrn_ has joined #openstack-infra14:59
*** zul has joined #openstack-infra14:59
*** sarob has joined #openstack-infra14:59
*** yfried_ is now known as yfried|afk15:00
jheskethanteaya: after the big example (or find 'start' inline)15:00
mordredgreghaynes: aroo?15:00
*** Hal has joined #openstack-infra15:01
*** asselin has joined #openstack-infra15:01
greghaynesmordred: the ssh key only having the asn.1 data15:01
mordredgreghaynes: in vagrant-devstack15:01
*** ChuckC_ has joined #openstack-infra15:01
*** Hal is now known as Guest5166615:01
*** ociuhandu has joined #openstack-infra15:01
mordredgreghaynes: I reached the point where not having a local devstack was detrimental to me getting anything done15:02
*** dkranz has quit IRC15:04
*** tiswanso has quit IRC15:04
fungiAJaeger: oh, after your discovery of whichever package that was that somehow failed to be listed in the pypi json and so bandersnatch helpfully deleted it on all our mirrors, i did a full refresh on all of them overnight. should be there again now15:06
*** fhubik has quit IRC15:06
*** mase_x200 has quit IRC15:06
*** thinrichs has joined #openstack-infra15:06
*** nelsnelson has joined #openstack-infra15:07
*** yfried|afk is now known as yfried_15:07
fungidstufft: thanks again for fixing that yesterday15:07
*** dkranz has joined #openstack-infra15:07
*** arxcruz has quit IRC15:08
*** davideagnello has joined #openstack-infra15:09
jeblairfungi, zaro: i just spun up the .19 war on a trusty vm with our puppet and did not see the problem15:09
jeblairfungi, zaro: it did start15:09
zxiiroyou folks gonna be at OpenStack summit?15:10
fungizxiiro: you mean all 432 people in this channel?15:10
openstackgerritKyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot
zxiirosure :)15:10
zxiiroopen question15:10
fungijeblair: yeah, one of zaro's theories is that there's something about the puppet manifest's routine to unpack the libraries from the war which isn't doing what it should15:11
*** jamesmcarthur has quit IRC15:11
*** jamesmcarthur has joined #openstack-infra15:11
*** xylan_kong has joined #openstack-infra15:12
*** tiswanso has joined #openstack-infra15:12
fungijeblair: which might also explain why rolling it back to the war we had been using worked... it might have still failed to unpack something correctly, but what was already present had been unpacked properly from that war long-ish ago15:12
*** annegentle has quit IRC15:12
*** ivar-lazzaro has joined #openstack-infra15:12
jeblairfungi, zaro: hrm.  i did it first with .17, then edited the manifest to install .1915:12
fungiokay, so the puppet manifest is even tested working for you. mostly rules out that theory too15:12
jeblairso it should be _roughly_ similar... though i suppose the prod server was probably built with << .1715:13
jklareAJaeger: clarkb: hi, would be great if you could recheck this when you have a minute (we just talked about it in our core meeting and would love to move forward)15:13
mordredsdague: ok - more questions - devstack-vagrant worked -so thank you15:13
*** davideagnello has quit IRC15:13
mordredsdague: is there  a "good" way to bind-mount directories from my host computer into the vagrant vm? like, if I wanted a local git repo in there?15:14
*** ildikov has joined #openstack-infra15:14
*** sabeen1 has joined #openstack-infra15:14
*** ivar-lazzaro has quit IRC15:14
jeblairfungi: yeah, i'm running out of ideas other than 'futz with it in prod' :/15:14
*** afazekas has quit IRC15:15
*** ivar-lazzaro has joined #openstack-infra15:15
*** salv-orlando has joined #openstack-infra15:16
jeblairzaro: did you give puppet a real gpg key for the contactstore?  that's probably necessary, i think it parses it and loads it in.  should be fine to use any valid public gpg key15:16
jeblairzaro: (it wants ascii armored public key export)15:16
sdaguemordred: local_openstack_tree: /home/sdague/code/openstack15:17
sdaguein config.yaml15:17
fungijeblair: zaro: maybe we can neuter the puppet manifest to not run a lucene reindex (this minor change difference in the war shouldn't require one i wouldn't think?) to make turn-around on testing and rollback a lot faster15:17
sdaguethat will mount that up to /home/vagrant/openstack15:17
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Bump up delorean to allow for openstack-dashboard installation
sdagueI should probably add magic that puts that also in /home/stack, but haven't yet15:18
jeblairfungi: yeah, i think we could do that temporarily15:18
mordredsdague: so - does that expect flat structure or group/repo structure?15:18
sdagueit's just mounting up the entire subtree15:18
sdaguebasically it can be anything you want it to be15:19
sdagueit's a directory locally that you would like to be /home/vagrant/openstack in the guest15:19
mordredok. so it's not necessarily going to tell devstack to use repos from there15:19
*** dannywilson has joined #openstack-infra15:19
sdaguemordred: oh, yes, it will be15:19
*** dizquierdo has quit IRC15:19
*** dguitarbite has quit IRC15:19
sdaguesorry, I never built that next level of logic to automatically point devstack at it15:20
mordredyah - so - does /home/vagrant/openstack want nova to be in /home/vagrant/openstack/openstack/nova or /home/vagrant/openstack/nova ?15:20
jeblairfungi: so maybe we can stage this so that it's easy to try next time it gets stuck.  so we don't schedule downtime, but if downtime schedules us, we can slip it in?15:20
mordredsdague: wait- I'm confused15:20
sdaguemordred: ... what are you trying to do15:20
mordredsdague: I actually don't need devstack to know anything about these repos15:20
sdaguemaybe I'm confused15:20
*** rossella_s has joined #openstack-infra15:20
mordredsdague: I have a shade repo, and I want to hack on it adjacent to a running devstack15:20
mordredI just want to make sure I don't mess something else up15:21
sdagueyep, you can use this then15:21
fungijeblair: oh, good point, that can be done in a puppet dev env, so don't actually need to temporarily patch it out of master15:21
mordredsdague: thanks!15:21
sdaguenote... it mounts into the guest as the vagrant user15:21
sdaguewhich is sort of how vagrant is designed15:21
jeblairfungi: ah yeah -- and if it succeeds, you'll need to disable puppet; merge prod to match the environment; enable puppet15:22
mordredsdague: sweet. I think this is going to make my life much easier15:22
*** yfried_ has quit IRC15:22
sdagueso if you pip install -e it, I think you can then just hack from host and life is good15:22
fungijeblair: right, getting that set up in my "fungi" dev env on the puppetmaster now15:22
*** marun has joined #openstack-infra15:22
mordredsdague: last question - I did a vagrant reload to get that mount to happen - and the moutn happened and was happy15:23
jeblairfungi: probably don't need to even merge the neuter change, only the version bump15:23
*** dboik_ has joined #openstack-infra15:23
mordredsdague: but openstack is no longer running15:23
fungijeblair: i concur. that's what i'm planning for15:23
*** scheuran has quit IRC15:23
mordredsdague: I want to run something in the devstack dir, right?15:23
*** sabeen1 has quit IRC15:23
*** dannywilson has quit IRC15:23
*** dkranz has quit IRC15:23
sdaguemordred: just again15:23
mordredsdague: cool15:24
sdaguereload takes the guest down / up15:24
mordredsdague: as the stack user, yeah?15:24
mordredlook at all the things I'm learning today :)15:24
*** dtantsur is now known as dtantsur|afk15:24
sdaguethere are many warts in this whole system, because it's sort of a "works for me" state, so suggestions on changes to make it better for other folks are very welcome15:24
*** dannywilson has joined #openstack-infra15:25
jeblairfungi: i'm heading into more intense conferencing now; ttyl15:25
*** thinrichs has quit IRC15:25
mordredsdague: cool. well, it works well for me at the moment15:25
fungihrm... jeblair how do you go about testing other modules besides openstack_project using a dev-env?15:25
sdagueI appreciate the new eyes looking at this and asking good questions15:25
fungijeblair: i'll get it worked out and let you know later. happy conferencing15:25
mordredsdague: with the network bridging - is it possible for me to point a REST client at the thing in the vagrant vm from my host?15:25
jeblairfungi: something something environemnt module path mumble?15:25
sdaguemordred: yes15:26
*** otter768 has joined #openstack-infra15:26
fungijeblair: thanks ;)15:26
sdaguethat's what I do, I network bridge it and specify a stable mac15:26
*** dboik has quit IRC15:26
*** sabeen1 has joined #openstack-infra15:26
sdaguethen it means it always ends up on the same IP on my home network15:26
sdagueso it's accessible from other machines on the network as well15:27
rcarrillocruzyah, that or forward the port from the host to the guest15:27
mordredah - so I'm simpler - there is no local network really that's usable taht way - localhost is fine15:27
sdagueI think even without the bridge though, you should be able to hit it on the hostonly network from your host15:27
rcarrillocruzbut i agree with sdague, bridgining is easier and faster15:27
mordredso I may just need to do port forwarding15:27
*** hdd_ has quit IRC15:27
* mordred is imagining using this on a plane too15:28
mordredso expecting the vm to get an IP from an environment is probably bad15:28
*** MaxV has quit IRC15:28
sdaguemordred: yeh, so for the plane case, I think just hit the host only network15:28
*** sks has quit IRC15:28
*** MaxV has joined #openstack-infra15:28
mordredI'll figure that out next15:28
sdaguebut... I don't do it that way, so it might require some futzing15:29
mordredssh-ing to the vm is not particularly onerous :)15:29
phschwartzGiving you guys a heads up, I found an issue with a commit to JJB that was done in the past 4 months that breaks usage of versions of Jenkins < 1.575. I am testing a fix, but so you guys know incase anyone else has an issue.15:29
*** spredzy_ is now known as spredzy_|afk15:29
*** harlowja_at_home has joined #openstack-infra15:29
mordredphschwartz: cool, thanks!15:30
*** thinrichs has joined #openstack-infra15:30
sdagueoh, I think "use_bridge: false"15:30
sdaguewill give you what you want15:30
sdaguesomeone else added that bit15:30
*** otter768 has quit IRC15:30
*** adrian_otto has joined #openstack-infra15:31
*** MaxV has quit IRC15:32
*** tjones1 has quit IRC15:32
adrian_ottogood morning. I am working on creating a stabe/kilo branch in Gerrit for Magnum, and I am looking for the "Admin", "Projects" menu while signed in, and I'm unable to find it. Is there a different way to create branches in Gerrit?15:33
anteayajhesketh: thank you15:34
adrian_ottoI can't find the word "Admin" in any of the Gerrit menus15:34
fungiadrian_otto: can you link to the documentation where it says there's a menu named "admin"? you're the second person i've seen ask about that15:34
adrian_ottofungi: I actually found it in the drivers docs… here: in the "End of Release" section15:35
*** nelsnelson has quit IRC15:35
*** rkukura has quit IRC15:36
fungiadrian_otto: assuming your permissions are correct, you select projects->list at the top of the webui, then find and click on the repo you want by using the filter there, then go to "branches" linked at the top of the page15:36
adrian_ottoYes, I get a list of branches15:36
fungiadrian_otto: i'll try to update the drivers guide for better clarity15:36
adrian_ottobut I don't see a way to add one15:36
*** dboik_ has quit IRC15:36
adrian_ottoI got to here:,branches15:36
*** dboik has joined #openstack-infra15:37
adrian_ottobut for me, no action buttons are present on that page15:37
fungiadrian_otto: if your account is a member of a group with "create" privileges in the refs/heads/* section of the acl for that repo, then there should be a "create branch" button and some fields to enter a branch name and initial revision15:37
*** dkranz has joined #openstack-infra15:37
*** sputnik13 has quit IRC15:37
adrian_ottoI am a member of magnum-core, but maybe I need to extend the ACL?15:38
*** armax has joined #openstack-infra15:39
fungiadrian_otto: yeah, looks like you don't have any groups delegated for branch creation. add a line like "create = group magnum-milestone" to your acl file in the openstack-infra/project-config repo15:39
mordredsdague: WOOT15:39
mordredsdague: works like a charm15:39
fungiadrian_otto: it'll need to be in alpha order between the abandon and label lines in the refs/heads/* section15:39
adrian_ottofungi: ok, will give that a try.15:39
fungiadrian_otto: also, only gerrit admins can delete branches, so you should probably restrict create to a subset of your core team to avoid accidental branch creation15:40
fungiadrian_otto: which is why i suggested your magnum-milestone group there15:40
*** mpavone has quit IRC15:40
adrian_ottofungi: there is a magnum-milestone group, so I can use that I suppose15:40
*** Swami has joined #openstack-infra15:41
*** vlaza has quit IRC15:41
*** flashgordon is now known as jogo15:42
*** jamesmcarthur has quit IRC15:42
*** jamesmcarthur has joined #openstack-infra15:43
*** jroll has quit IRC15:44
ttxjeblair: random Depends-On feature: make sure a backport patch can't merge in a branch before the master patch gets in.15:44
ttxwould remove a lot of release / security manual checking15:45
fungii think that would require extending the depends-on syntax to be optionally project and branch specific15:45
fungibut we've seen other potential use cases for that as well15:46
*** hasharMeeting has quit IRC15:46
*** hashar has joined #openstack-infra15:46
fungifor example, depending on a master branch patch where the stable branch patch of the same change-id has been abandoned15:46
*** jroll has joined #openstack-infra15:46
*** sdake has joined #openstack-infra15:46
*** hashar is now known as hasharAway15:47
fungior when a change to another project using the same change-id has been abandoned or isn't a legitimate dependency of the change proposed15:47
openstackgerritDavide Guerri proposed openstack-infra/shade: WiP: Add Neutron/Nova Floating IP support
*** hasharAway is now known as hasharAway-15:47
openstackgerritAdrian Otto proposed openstack-infra/project-config: Added branch creation to Magnum ACL
adrian_ottofungi: okay, proposed that edit15:47
fungiadrian_otto: yep, lgtm15:47
*** yamamoto has quit IRC15:48
*** sabeen1 has left #openstack-infra15:48
*** nelsnelson has joined #openstack-infra15:49
adrian_ottothanks fungi for the guidance15:49
*** sabeen1 has joined #openstack-infra15:49
fungiadrian_otto: any time!15:49
*** yamamoto has joined #openstack-infra15:49
*** kgiusti has left #openstack-infra15:51
openstackgerritKyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot
*** jroll has quit IRC15:53
*** jroll has joined #openstack-infra15:53
*** salv-orlando has quit IRC15:53
*** e0ne is now known as e0ne_15:53
*** e0ne_ is now known as e0ne15:54
*** tjones1 has joined #openstack-infra15:54
anteayamorning zaro15:55
*** gyee has joined #openstack-infra15:55
zarofungi, jeblair : upate to .19 does not require a reindex.15:56
zarofungi: when are you thinking about trying the gerrit update?15:56
*** Sukhdev has joined #openstack-infra15:56
fungizaro: the next time stream-events hangs15:56
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct output path for infra-site index
fungiAJaeger: ^ see for details15:58
fungimordred: ^15:58
*** asselin has quit IRC15:58
zarofungi: ok.  i guessing i'm mising that key jeblair was refering on my VM to be able to get contact store to work.  I'm not sure how to create/get that key though.15:58
*** AJaeger_ has joined #openstack-infra16:00
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix publish-infra-docs-index
*** afazekas has joined #openstack-infra16:00
zaropelix: thanks for the rebase to 75514.  i ran my tests again but am not confident yet.16:00
AJaeger_fungi, the publish job was still wrong, I expect that change 177842 fixes it finally16:00
pleia2good morning16:01
*** david-ly_ is now known as david-lyle16:01
fungiAJaeger_: that looks identical (minus your typo) to teh change i was pointing out to you16:02
*** MaxV has joined #openstack-infra16:02
zaropelix: i got the same error I referenced in PS 19 on one of my test runs.16:02
*** nelsnelson has quit IRC16:03
fungizaro: install gnupg and then run `gpg --gen-key` and follow the prompts (picking defaults if you want) and after that's done do `gpg --armor --output some_file --export whatever_the_key_id_is` and then the contents of some_file are the key you want. alternatively just borrow someone's published key16:03
AJaeger_fungi: just came home and checked why it was not working - should have read backscroll...16:03
*** alexpilotti has joined #openstack-infra16:04
AJaeger_Ok, let me review yours ;)16:04
*** nelsnelson has joined #openstack-infra16:04
zaroanybody know why zuul test would fail when run with 'tox -epy27' but not fail when run with testr inside of the virtualenv?16:04
fungizaro: mine is linked at if you want to just use it as a test16:04
*** sergsh has quit IRC16:05
*** jlanoux has quit IRC16:05
zarofungi: thnaks16:05
*** ildikov has quit IRC16:06
*** dkranz has quit IRC16:06
*** Guest51666 has quit IRC16:06
AJaeger_fungi, ah, you beat my by two minutes ;)16:06
zarothe zuul test failure is "process-returncode [ multipart" with "returncode 1420"16:07
fungiAJaeger_: i guess i'm just fast like that16:07
fungiyou usually beat me to these with a wide margin16:07
*** MaxV has quit IRC16:07
*** dmellado has joined #openstack-infra16:08
AJaeger_fungi: you're really fast - and if we start at the same time, I guess you always beat me ;)16:08
AJaeger_fungi: you're often firefighting several fires at once ;)16:08
fungiAJaeger_: anyway, it's just as well. it was my mistake after all ;)16:08
pelixzaro: I was thinking it might be worth investing in writing some tests to exercise the parallel section of update_job either by mocking more of the python-jenkins api and letting it be called via threads or by creating a simple HTTP server from python and handling testing requests that way to shake out socket exceptions?16:08
AJaeger_fungi: we're getting it fixed ;)16:09
*** salv-orlando has joined #openstack-infra16:09
*** ChuckC_ has quit IRC16:09
*** ildikov has joined #openstack-infra16:09
*** mpaolino has quit IRC16:09
*** salv-orl_ has joined #openstack-infra16:10
*** devvesa has quit IRC16:10
*** rbradfor has joined #openstack-infra16:11
*** thinrichs has left #openstack-infra16:11
anteayamorning pleia216:11
*** SpamapS has quit IRC16:12
*** SpamapS has joined #openstack-infra16:12
*** tqtran has joined #openstack-infra16:12
*** salv-orlando has quit IRC16:13
*** AJaeger_ has quit IRC16:14
*** ildikov has quit IRC16:14
*** Sukhdev has quit IRC16:15
*** jcoufal has joined #openstack-infra16:15
*** sabeen2 has joined #openstack-infra16:15
*** sabeen1 has quit IRC16:15
*** unicell1 has quit IRC16:15
*** zul has quit IRC16:16
*** MaxV has joined #openstack-infra16:16
sdagueso is dfw in rax having issues this morning?16:17
*** dkliban has left #openstack-infra16:17
sdague is the second time I've had a patch this morning get killed there16:17
*** dkranz has joined #openstack-infra16:19
sdaguefungi: oh, wait, those are *our* mirrors16:19
sdagueany idea why our mirrors are broken?16:19
anteayasdague: your prior message was the only complaint I have seen on our mirrors16:20
anteaya2015-04-27T14:50:32  <sdague> grr... pypi mirrors in rax randomly dropping connections16:20
samueldmqhi, I want to use the job defined here ( )16:20
samueldmqthe only thing I need to do is to include it on the experimental part of the project I want to run it agains at zuul/layout, right ?16:21
sdagueso, I've had at least 3 test fails this morning from - disconnecting me16:21
*** amotoki_ has joined #openstack-infra16:21
openstackgerritMerged openstack-infra/project-config: Added branch creation to Magnum ACL
zaropelix: i think the latter might be more useful that we we can simply just add update and delete tests as well.16:21
*** amotoki has quit IRC16:21
*** jistr has quit IRC16:21
*** amotoki_ is now known as amotoki16:21
zaropelix: something like this?
*** armax has quit IRC16:22
anteayasamueldmq: it is on the tempest repo right now:
anteayasamueldmq: which repo do you want it on?16:23
openstackgerritKyle Mestery proposed openstack-infra/project-config: Add #opendaylight-neutron to gerritbot
samueldmqanteaya, devstack16:23
fungisdague: load average looks high-ish there
zaropelix: regarding that change, '--workers' param is a noop for 'test' cmd right?16:23
zaropelix: should it be a sub-menu of 'test'?16:23
samueldmqanteaya, so that we'll make changes on devstack, and then run it against that16:23
*** armax has joined #openstack-infra16:23
*** viktors is now known as viktors|afk16:23
samueldmqanteaya, using 'check experimental'16:24
anteayasamueldmq: well it might be a good idea to ask mtreinish if he wants that16:24
anteayasamueldmq: and then yes, adding the job name to the tempest experimental pipeline is the code to make it happen16:24
anteayasorry devstack experimental pipeline16:24
samueldmqanteaya, right, but for now we just want tot test16:25
samueldmqanteaya, yes, experimental pipeline16:25
adrian_ottohey fungi, that patch merged but even after logging out of Gerrit, and logging back in, I'm still not seeing a way to add a branch on the,branches page. Is there more that needs to be done?16:25
fungisdague: oh, exciting. bandersnatch seems to be running there in a loop again. i wonder if the way i file locked during my refresh last night didn't prevent it from conflicting somehow. i'll disable the cron job and get one loop going16:25
fungiadrian_otto: it takes about 30 minutes or so for that to get applied16:26
adrian_ottooh, thanks. I'll check back on that later then.16:26
sdagueyeh, I've found fails 5 and 6 on that16:26
fungisdague: just dfw? it's doing it on all four mirrors from what i can see16:26
*** ociuhandu has quit IRC16:26
sdagueso, I only seem to notice it in DFW16:26
fungimaybe dfw is just on an especially unhappy compute node16:27
*** liam__ has joined #openstack-infra16:27
*** mpaolino has joined #openstack-infra16:27
sdagueyeh, it looks like DFW is the only one that we're actually failing on16:28
sdaguethose are basically all job fails16:28
* anteaya is going to get a walk in16:28
*** hyakuhei has joined #openstack-infra16:29
openstackgerritDavide Guerri proposed openstack-infra/shade: WiP: Add Neutron/Nova Floating IP support
*** dboik_ has joined #openstack-infra16:30
*** dizquierdo has joined #openstack-infra16:30
*** markvan has quit IRC16:32
*** sabeen1 has joined #openstack-infra16:32
*** tiswanso has quit IRC16:32
*** markvan has joined #openstack-infra16:32
*** tiswanso_ has joined #openstack-infra16:33
fungiokay, puppet agent disabled, crontab edited and current bandersnatch runs killed on all our pypi mirrors, then fresh resyncs started in a root screen session on each of them16:33
fungiin a few hours i'll check back in on them to see whether they've finished16:34
*** tonytan4ever has quit IRC16:34
fungiand reenable cron/puppet if so16:34
*** ociuhandu has joined #openstack-infra16:34
*** liam__ has quit IRC16:34
*** dboik has quit IRC16:34
*** sabeen2 has quit IRC16:35
*** dhritishikhar_ has quit IRC16:35
*** gordc is now known as gordc_afk16:35
*** afazekas has quit IRC16:35
*** dhritishikhar_ has joined #openstack-infra16:36
*** shashankhegde has joined #openstack-infra16:36
*** afazekas has joined #openstack-infra16:36
*** ashleighfarnham has joined #openstack-infra16:36
openstackgerritTim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient
openstackgerritTim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient
*** hyakuhei has quit IRC16:40
*** ashleighfarnham has quit IRC16:41
clarkbfungi what prompted the refresh last night?16:41
*** ssam2 has quit IRC16:42
openstackgerritTim Hinrichs proposed openstack-infra/project-config: ACL changes for python-congressclient
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/project-config: Add experimental Identity v3 jobs to devstack
*** hyakuhei has joined #openstack-infra16:43
samueldmqmorganfainberg, experimental devstack identity v3 jobs for devstack ^16:44
samueldmqdolphm, cc ^16:44
fungiclarkb: dstufft thinks the glusterfs backend for pypi had a fit, but anyway we noticed at least one package bandersnatch deleted from all our mirrors for not being listed in the pypi json. he corrected the problem at the pypi side but that's not covered by a new serial so bandersnatch continued to be out of the loop16:44
dolphmsamueldmq: \0/16:44
*** MaxV has quit IRC16:44
samueldmqdolphm, well, that's andreaf good job !16:45
dstufftI'm not a big fun of glusterfs16:45
fungii can see why16:45
*** ashleighfarnham has joined #openstack-infra16:45
openstackgerrityolanda.robla proposed openstack-infra/system-config: Move server outside of puppetmaster class
samueldmqdolphm, the goal now is to get morganfainberg's changes on devstack (disabling v2 for tests) and run 'check experimental' on them o/16:45
dstufftsometimes our web nodes randomly lose connectivity, where attempting to CD or anything into the directory returns a not exist16:45
dstufftand the only way to fix it is to restart the glusterfs netfs service (sometimes... once I had to restart the whole server)16:46
morganfainbergsamueldmq: thanks16:46
morganfainbergsamueldmq: but there is likey more needed than just that: ds-gate updates among other things16:46
sdaguefungi: where should I file a bug here to track the ER query?16:47
samueldmqmorganfainberg, I thought it was only that, I asked anteaya and she confirmed16:48
samueldmqmorganfainberg, the job template is already created .. so isn't jsut that ?16:48
morganfainbergsamueldmq: oh hm... i am not sure that will do what you think it does16:49
*** armax has quit IRC16:49
*** derekh has quit IRC16:49
samueldmqmorganfainberg, that makes tempest to use v3 tokens to its tests16:49
*** hdd has joined #openstack-infra16:50
samueldmqmorganfainberg, so that we can make the devstack changes (disabling v2 for eg) and running 'check experimental'16:50
samueldmqmorganfainberg, that's what I think16:50
openstackgerrityolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options
*** _nadya_ has joined #openstack-infra16:50
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: Add query for pypi mirrors disconnecting runs
sdaguemtreinish: thanks16:50
morganfainbergsamueldmq: that only changes tempest to use v316:50
morganfainbergsamueldmq: that doesn't actually do much else.16:50
morganfainbergsamueldmq: a bunch of other things will need to land16:51
sdaguethat's 144 fails in the last 7 days, so that should be position 2 in elastic recheck16:51
*** Sukhdev has joined #openstack-infra16:51
samueldmqmorganfainberg, yeah, and after that, we just need to change devstack, no ?16:51
morganfainbergsamueldmq: possibly more16:52
morganfainbergwe might want an extra DS-GATE toggle to disable v216:52
morganfainbergbut this is a good enough start16:52
fungisdague: all dfw? if so, then this is likely unrelated to the bandersnatch refresh from last night and more likely that there's something wrong in dfw, perhaps with the compute node running that instance16:52
*** afazekas has quit IRC16:52
samueldmqmorganfainberg, yeah, I am planning to use this just to test the devstakc changes16:52
*** deepakcs has joined #openstack-infra16:53
*** unicell has joined #openstack-infra16:53
sdaguefungi: from scrolling through that list it all looks like dfw16:53
sdague144 fails16:53
sdaguemaybe it's 142 fails and 2 succeeds actually on that query16:53
sdaguebut everything is dfw16:53
samueldmqmorganfainberg, ++ for a new devstack gate with v2 disabled16:54
sdaguefungi: query, logstash url, and example fail fail in there16:54
openstackLaunchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New]16:54
fungisdague: yeah, this seems like maybe network issues, perhaps at the hypervisor level (we've seen this before when a neighbor on the same compute node is flooding/flooded by network traffic beyond what the physical interfaces can accommodate)16:54
*** unicell has quit IRC16:54
*** unicell has joined #openstack-infra16:54
cineramapleia2: do we have a list handy of what repositories the translate jobs run on? i want to create a list of what locales we're currently using16:54
pleia2cinerama: I believe it's in transifex, let's see...16:55
*** unicell has quit IRC16:55
*** unicell has joined #openstack-infra16:55
sdagueanyway, I need to walk away for a bit16:55
cineramapleia2: cool thanks16:56
pleia2cinerama: currently, I believe when we need to add a new project or language, it's all done via the transifex interface (new projects are a customer support ticket)16:56
pleia2so sadly no spiffy config file to pull from16:56
cineramapleia2: yeah we're going to need to add the languages we use manually on zanata for now. i also need to check whether the locale names we're using match up with what zanata has16:57
pleia2cinerama: ah, good point16:57
*** vhoward has joined #openstack-infra16:58
*** erlon has joined #openstack-infra16:58
cineramapleia2: we can create a mapping in the zanata.xml we're autogenerating from zanata name to other name so our files in the repo will still have the same location16:59
pleia2cinerama: good to know!17:00
cineramapleia2: i was thinking i will just grab all the current locales and have that section as boilerplate in the xml17:00
* pleia2 nods17:00
pleia2I don't actually know where the "73 languages" are listed in transifex17:00
*** afazekas has joined #openstack-infra17:00
pleia2ah, on each project page you can "show all languages"17:01
*** harlowja_at_home has quit IRC17:01
*** dhritishikhar_ has quit IRC17:01
*** dhritishikhar_ has joined #openstack-infra17:01
*** patrickeast has joined #openstack-infra17:02
*** bswartz has quit IRC17:03
*** alexpilotti has quit IRC17:03
*** Ala has quit IRC17:04
*** davideagnello has joined #openstack-infra17:05
*** Krinkle|detached is now known as Krinkle17:06
*** yamahata has joined #openstack-infra17:07
*** dalgaaf has quit IRC17:07
*** wuhg has quit IRC17:09
fungisdague: i've opened a fanatical support ticket for pypi.dfw tcp read timeouts. hopefully that turns something up. logs on the instance itself don't show any obvious problems17:10
sdaguefungi: coolio, thanks17:10
fungithanks for spotting the issue17:10
clarkbfungi: sdague is it reproduceable from your local end?17:10
*** tnovacik has quit IRC17:11
sdagueclarkb: *shrug*?17:11
sdaguethe bug has what I have, a logstash signature, over 140 failed jobs, and an example to what a fail looks like17:11
fungiclarkb: it seems to be needle-in-a-haystack occurrences. e.g. one random pip download out of the many that happen during any given job17:11
sdagueyeh, it's not the same package over and over again17:11
fungiso we'd need to do some heavy looping of requests to it to reproduce manually, i expect17:12
clarkbSpamapS: had a question about late friday, not sure you saw it. Any idea why line 165 uses the __get__ lookup of func but 169 does not? is that a bug?17:12
clarkbfungi: sdague gotcha17:12
clarkbfungi: sdague one thing I would check is the eth0 bw useage17:13
clarkbfungi: sdague the git haproxy nodes saw tcp resets when they hit their bw max17:13
clarkbI think its 400Mbps for the 8GB Performance flavor17:13
tqtran following procedure to create a new repo but its not passing. Can someone guide me in the right direction?17:13
*** ociuhandu has quit IRC17:13
sdagueclarkb: it's pypi mirrors17:13
clarkbsdague: I know, I am saying it could be bw limiting17:14
mordredone could imagine those hitting BW caps17:14
marunApologies for having asked this before, but what's the timeline on storyboard usurping launchpad?17:14
*** ociuhandu has joined #openstack-infra17:14
sdagueclarkb: is that our side, or theirs?17:14
clarkbsdague: rax does the limiting for us17:14
clarkbsdague: and it results in really weird tcp resets where our side doesn't send it17:14
marunWe're looking at some process changes in Neutron around feature requests and I'm wondering if we should be considering the process storyboard will impose.17:14
mordredmarun: unknown - there is a larger question floating at the moment as to whether being in teh business of writing storyboard is the business we shoudl be in17:14
clarkbtqtran: is the errors, looks like you added a few jobs to the zuul layout that are not defined in jenkins job builder17:15
sdagueclarkb: ok17:15
dougwiganteaya: (or anyone) - sanity check, is it correct to assume that using stackforge to implement a third-party CI is not allowed?17:15
asselin_dougwig, correct17:15
mordredmarun: I supposed I should say "I threw a wrench in the works by raising that question" - in any case - I do not believe there is any chance we'll be off of launchpad before tokyo17:15
marundougwig: if you have to ask... ;)17:15
sdagueok, stepping away for a bit, hopefully someone can look into that, because I'm recheck grinding a tempest patch series because of it17:16
marunmordred: ok, thanks for the info17:16
asselin_dougwig, we're putting it here: openstack-infra/puppet-openstackci17:16
mordredmarun: sure nuff! I'm pretty sure it'll be an important topic for discussion in vancouver17:16
clarkbsdague: looking at I don't think it is BW rate limiting17:16
marunmordred: ++ for being wary of scope creep17:16
clarkbsdague: we are no where near the 400Mbps I expect to get throttled at17:16
*** shashankhegde has quit IRC17:16
dougwigasselin_: i'm more noodling if there's some hybrid possible where one group runs a common jenkins/zuul/nodepool, since maintaining that is where all the failures seem to occur.  overlays and virtualization mean that i don't need anything onsite.17:16
*** EmilienM is now known as EmilienM|afk17:17
dougwigasselin_: as opposed to us all running the same scripts.17:17
fungiclarkb: also it looks like socket timeouts, not reset by peer17:17
clarkbfungi: gotcha17:17
openstackgerritJoe Gordon proposed openstack-infra/project-config: Reduce number of services running in largeops-src
*** afazekas has quit IRC17:18
fungia friend just ran into
fungilooks super fun17:18
asselin_dougwig, that's  an interesting idea. main issue I see is regarding trust/security...17:18
clarkbdougwig: I can imagine that convincing corporate network security to allow ssh into wherever your slaves are will be near impossible17:18
clarkbasselin_: yup thats where I expect to see the problems too17:18
openstackgerritDoug Hellmann proposed openstack-infra/elastic-recheck: Update query for bug 1403284
openstackbug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] - Assigned to Joe Gordon (jogo)17:19
*** xyang1 has joined #openstack-infra17:19
*** achanda has joined #openstack-infra17:20
dougwigi got around that by having my slaves the same as infra: single-use devstack nodes.  on AWS, no less, and they use standard hooks to spin up a virtual appliance at the same place (which could be spun up across the internet somewhere via a hook). it'd literally be 100 lines of yaml/script in a stackforge project to replicate, with no security concerns that I17:21
dougwigdon't already have.  note that I am NOT suggesting community resources go to companies here, just exploring if we can maybe get some common maintenance around the parts not relevant to the test at hand.17:21
*** sputnik13 has joined #openstack-infra17:21
dhellmannfungi, clarkb, anteaya: can I get you to look at for us? problems with that job are blocking us from merging an unrelated change in oslo.messaging that we need for the stable release17:21
clarkbdougwig: most third party CI requires special hardware local to company right?17:21
*** e0ne has quit IRC17:21
clarkbdougwig: in your example is there anything preventing you from just testing upstream? (possibly non open source code?)17:21
dougwigclarkb: i'd imagine that in neutron land, overlay networks and virtualization can get around that.  in others, a DMZ piece of hardware.17:22
dougwigclarkb: do you mean me testing master vanilla?  no.  do you mean me testing in the same environment as master?  no.  not sure i'm parsing that properly.17:22
asselin_dougwig, what are you testing?17:23
clarkbdougwig: I mean if you are just going to use virtual machines anyway (eg no hardware dependency) why not run your tests with our nodepool/jenkins/zuul17:23
clarkbdougwig: since we already provide this service you are looking for17:23
dougwigclarkb: that was my opening question: "sanity check, is it correct to assume that using stackforge to implement a third-party CI is not allowed?"  :)17:23
*** sergmelikyan has joined #openstack-infra17:23
clarkbdougwig: I missed that17:23
clarkbdougwig: glusterfs and ceph and drbd for cinder are all doing it17:24
clarkbthough ceph isn't in stackforge thats in tree iirc17:24
dhellmannfungi: thanks17:24
dougwigasselin_: a10 appliance with lbaas is what we're testing, to answer your question.17:24
clarkbbut glusterfs and drbd have devstack plugins in stackforge that test their thing against openstack upstream17:24
clarkbdougwig: the key there is they are all open source17:24
sergmelikyanHi folks, what is timeout to sync our mirrors with PyPI? We released new version of YAQL library in order to fix critical issue but it is already 3h are not found in gate:
fungidougwig: your question was unclear to me too... totally depends on what you mean by "using stackforge" and "implement a third-party ci17:24
clarkband don't rely on special hardware17:24
*** mpaolino has quit IRC17:24
*** deepakcs has quit IRC17:24
*** jamesmcarthur has quit IRC17:24
dougwigahh, yes, i meant to explore if closed-source third-party can somehow figure out a way to not replicate all the parts.17:25
fungidougwig: how are you attaching your a10 appliance to amazon ec2 instances?17:25
openstackgerritDoug Hellmann proposed openstack-infra/elastic-recheck: Make the query for bug 1403284 more generic
openstackbug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] - Assigned to Joe Gordon (jogo)17:26
*** salv-orlando has joined #openstack-infra17:26
dougwigfungi: it just needs an ip/user/password to connect anywhere.  today, i'm spinning up a soft appliance in parallel with the devstack node.  it doesn't need hardware.17:26
*** jamesmcarthur has joined #openstack-infra17:26
openstackgerritMatt Riedemann proposed openstack/requirements: add analayzer
*** otter768 has joined #openstack-infra17:27
*** salv-orl_ has quit IRC17:27
mriedemsdague: ^ backport to stable/juno17:28
fungidougwig: mostly trying to figure out if you're describing something like (i.e. hosting within stackforge the scripts and configuration which make up your ci)17:28
fungidougwig: or whether you're talking about having a third-party ci report on changes proposed to a repo in stackforge17:28
SpamapSclarkb: regarding the cache_on_arguments thing, IIRC, it has to do with binding.. I didn't take notes when I was making it work unfortunately.17:28
*** dhritishikhar_ has quit IRC17:29
*** dhritishikhar_ has joined #openstack-infra17:29
*** mpaolino has joined #openstack-infra17:29
clarkbSpamapS: right I htink you want the __get__ lookup to make sure you always get the correct func regardless of inheritance, so I am wondering why invalidate doesn't do the same17:29
*** marun has quit IRC17:30
fungidougwig: so i have no idea still what you mean by "find a way not to replicate all the parts"17:30
dougwigfungi: actually, i'm wondering if it'd provide stability gains if third-party jobs could use a common jenkins/zuul/nodepool/devstack-gate, with hooks to connect in their stuff, rather than everyone running all of that stuff.17:30
SpamapSclarkb: it has something to do with the fact that invalidate happens after binding.17:30
*** yfried_ has joined #openstack-infra17:30
fungidougwig: oh, so you mean multiple third parties connecting their workers into a centrally-run ci system?17:31
SpamapSclarkb: morganfainberg pointed out that we also should memorize the wrap in invalidate since we already have func bound.. would save re-wrapping inside dogpile.cache.cache_on_arguments every time invalidate gets called17:31
*** otter768 has quit IRC17:31
dougwigfungi: right.  instead of project-config, third-party-config, or whatnot.17:31
clarkbSpamapS: oh! I see ya that would explain it since func should be correct if its post binding17:31
fungidougwig: one challenge you're going to find is that currently zuul is only capable of leaving one vote on a proposed patchset. it can't leave votes under multiple accounts. so all jobs running via that coop zuul would have to share a single vote17:32
SpamapSclarkb: but it's entirely possible that we should be doing a __get__ in invalidate too.. I can make a test right now by extending OpenStackCloud ..17:32
clarkbSpamapS: I think __get__ will work for invalidate it just isn't necessary if your note about binding is correct17:32
*** notnownikki has quit IRC17:32
clarkbSpamapS: whereas it would be ncessary on line 16517:32
morganfainbergSpamapS: you could use functools partial to solve the issue with the late bind17:32
SpamapSclarkb: I went a little cross-eyed writing _cache_on_arguments so it is quite helpful to have more eyes on it.17:33
openstackgerritMerged openstack-infra/infra-manual: Reverse dependency relationship in project creation
fungidougwig: we've discussed the possibility of allowing third parties to slave their custom workers to but at this point any planning toward that capability probably needs to happen as part of the zuul v3 redesign17:33
SpamapSthe whole thing has soured me on the pattern really.17:33
SpamapSI'm quite inclined to just write my own cache_on_arguments method that doesn't have to bend over backwards like that.17:34
clarkbSpamapS: ya it is a bit complicated to follow17:34
*** sdake_ has joined #openstack-infra17:34
greghaynesah, I see you are memoizing in a decorator17:34
clarkbgreghaynes: ya17:34
greghaynesthat is how that usually ends17:34
greghaynespain and suffering17:34
*** mattfarina has quit IRC17:34
dougwigfungi: i'm not even sure you need to slave their workers, since many just run dsvm nodes with hooks.  though maybe to spin up a pool in a different cloud.17:34
SpamapSclarkb: I'd be ok with the complicated flow if I knew how it worked and could document it a little more. I'm kind of like "yeah sure it works see the tests passing?" :-/17:34
openstackgerritRussell Bryant proposed openstack/requirements: global-requirements: Add ovs
fungidougwig: then i'm back to not understanding why those don't just use our current workers connected to zuul.openstack.org17:35
SpamapSgreghaynes: all we need is a metaclass and we'll have a sufficiently complex piece of syntactic sugar that python devs everywhere will cower in fear of our decorators.17:35
dougwigfungi: because it's testing a closed-source solution.17:35
fungidougwig: if what they need is "to just run dsvm nodes with hooks" then we have that capability already17:35
morganfainbergSpamapS: use descriptors.17:35
SpamapSmorganfainberg: I think I am. Sort of. :)17:35
* greghaynes cowers in fear17:35
morganfainbergSpamapS: people will cower in the Java like syntax.17:36
SpamapSmorganfainberg: we have reached the frontier of my understanding, so I'm learning as I go here.17:36
fungidougwig: oh, so you mean testing software you can't publish. that sounds complicated to engineer as a shared solution17:36
*** sdake has quit IRC17:36
*** armax has joined #openstack-infra17:36
clarkbmorganfainberg: we are using descriptors17:36
fungidougwig: i'm not sure how you would manage to hide the source code for your closed-source software if it's being tested on a community-run ci17:36
clarkbmorganfainberg: but I think you mean implement descriptors such that the existing methods work without decoration?17:37
dougwigfungi: well, i'm testing the driver that is published (in neutron), and the appliance is a static widget. so my changes/maintenance are all in the CI plumbing itself. i don't think i'm alone in that.17:37
openstackgerritMerged openstack-infra/elastic-recheck: Add query for pypi mirrors disconnecting runs
dougwigfungi: i suspect we have a communication disconnect.  :)17:37
fungidougwig: yes, i have a feeling you and i mean completely different things with many of the words we're using17:37
*** marun has joined #openstack-infra17:37
clarkbmorganfainberg: sorry without the special sauce decoration. iirc dogpile uses decoration out of the box17:37
SpamapSclarkb: oh thats an interesting thought.17:38
greghaynesno, no17:38
fungidougwig: for me closed-source software is software for which the source code is not (legally) available to the general public. perhaps you're talking about non-free open-source software (i.e. software for which the source is public but which comes with restricted licenses that may hinder its reuse or derivation)17:39
openstackgerritMonty Taylor proposed openstack-infra/shade: Add a method for getting an endpoint
fungi(or hinder its redistribution)17:40
SpamapSfungi: dude, you're so Debian right now17:40
*** hdd has quit IRC17:41
fungiSpamapS: well, it's germane to the problem and solution being discussed17:41
fungidougwig: or are you talking about testing some free/open-source software's interactions with some other system (which may be not be generally available for various reasons)?17:41
*** shashankhegde has joined #openstack-infra17:41
SpamapSfungi: entirely, it was a compliment. :)17:43
fungidougwig: probably concrete examples of what you're trying to solve with your proposal would help clarify things, rather than speaking in abstract generalities17:43
fungiSpamapS: as long as i'm not too debian-devel mailing list kilothread flametastic i'm cool with that ;)17:44
dougwigfungi: i'm talking open-source openstack driver with closed-source backend, which tests every gerrit submission for a given openstack project (aka third-party CI.)  which is usually accomplished by running a copy of infra's setup, running vanilla devstack-gate, subbing in your driver in local.conf, and using a gate hook to create the connection to the17:45
dougwigproprietary backend.   it can all be run trivially on infra's setup today, but it's a closed-source backend. so all of the vendors run a copy of the infra setup, and almost all of our issues are unrelated to our drivers (it's almost all canonical and pypi download problems.)  i'd love to submit 10 lines of yaml to project-config and a gate hook in a repo and17:45
dougwigbe done.  so, i was just asking if something along those lines were feasible and lower-cost than the community pays for all the errors today.  i expect the answer is, "we don't want to touch proprietary stuff with a  10' pole", which is totally ok. i'm just pondering based on where my CI has been flaky, the cost that has incurrent for folks, and how can we17:45
dougwigreduce it.  the common puppet scripts are a great start, but that ongoing stuff that breaks isn't covered by that.17:45
*** sdake has joined #openstack-infra17:45
SpamapSclarkb: so I think what morganfainberg is suggesting is that we would make the methods we want to "decorate" non-data descriptors that would define a __get__ that would do the caching.. I think.17:45
dougwigsorry for the wall.  :)17:45
morganfainbergSpamapS: basically17:45
*** rkukura has joined #openstack-infra17:47
*** amotoki has quit IRC17:47
SpamapSmorganfainberg: and thats where you get into dogpile.cache having a metaclass that gives you the ability to do that more or less easily. Yes?17:47
morganfainbergThat was my thought. But it might be the wrong approach since I haven't tried it out.17:47
asselin_dougwig, if you can test your proprietary hardward using the -infra setup, just by adding a 10 line job, then that's probably what you should do (and what clarkb said earlier)17:47
clarkbSpamapS: morganfainberg gotcha so no decoration needed either just implement __get__ and __set__ such that caching happens17:47
asselin_dougwig, the assumption is that there are no security issues on your end.17:48
morganfainbergclarkb: aye.17:48
fungidougwig: okay, so you're asking how to test some otherwise inaccessible system/device's interaction with software running on workers coordinated by a central system... we already do a lot of that. i think it depends entirely on what kind of interaction you're needing to test17:48
clarkbasselin_: except that we explicitly don't want to be testing proprietary things17:48
*** sdake_ has quit IRC17:48
morganfainbergclarkb: that would be my approach probably now. But I haven't tried it. So beware of potential landmines.17:48
SpamapSwell.. you'd still decorate I think, but your decorator would just add the __get__ and __set__ to the method.17:48
clarkbasselin_: anything we run upstream we intend to have be reproduceable by any dev. eg if you get a jenkins -1 that shouldn't mean that I have to go and talk to $company to help you sort it out with them17:48
SpamapSSo it would all happen at the right time (parse time) and then you wouldn't have to rewrap at runtime like my method does.17:49
SpamapSI may dive down this rabbit hole a bit... it's quite interesting.17:49
*** marcusvrn_ has quit IRC17:49
asselin_clarkb, that assumes then that the propreitary hardware is publicly accessible to everyone17:49
* SpamapS wishing he hadn't taken a puck to the head yesterday...17:49
dougwigasselin_, fungi - right, what clarkb said is the reason we run a billion CI's now.17:50
dougwigi may be exaggerating slightly.17:50
*** dprince has quit IRC17:50
fungiasselin_: clarkb: dougwig: job running in the openstack ci making http calls to a random web service are probably a bit of a grey area17:50
clarkbfungi: we already don't allow PPAs and the like for the most part17:50
fungiif it needed custom credentials, we definitely wouldn't support it17:50
fungiwe do allow ppas17:51
fungiwe don't preconfigure them, but jobs can certainly modify the sources.list and install additional packages17:51
*** cdent has quit IRC17:51
clarkbthats true17:51
dougwigclarkb, fungi, asselin_: history tells us that infra is *way better* at running stable openstack CI's than the vendors. and there is a big cost to those false positives. is it possible/desired to leverage infra's awesomeness to help reduce that cost? that's my question.17:52
*** harlowja_away is now known as harlowja_17:52
*** mpaolino has quit IRC17:52
greghaynesSpamapS: have you looked at the functools lru_cache btw?17:52
mordredgreghaynes: I looked at it a little while ago before we went down the dogpile route17:54
fungidougwig: so, the reasons we envisioned third-party testing originall were for cases where your jobs specifically needed workers to locally interact with specific sorts of hardware or have access to non-public services or install restricted/licensed software or...17:54
greghayneseven if you dont want to just use it, the source is there and it should be close :)17:54
fungier, originally17:54
fungialso for the general case where the worker configuration was going to be too complex for us to maintain17:55
openstackgerritMerged openstack-infra/elastic-recheck: Update query for bug 1403284
openstackbug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] - Assigned to Joe Gordon (jogo)17:56
SpamapSgreghaynes: if we just want a global cache region, we can get that level of simplicity with dogpile.cache too.17:56
SpamapSgreghaynes: the complexity is that we have one _cache per OpenStackCloud.17:57
fungidougwig: asselin_: unfortunately a lot of projects who wanted to ensure various backends were thoroughly tested mis-stated their desires as wanting all the people responsible for those drivers and plugins to maintain separate external ci systems17:57
greghaynesSpamapS: ah!17:57
greghaynesSpamapS: namespaced_lru_cache17:57
clarkbgreghaynes: mordred for ansible synchronize do you know how to give a different dest_path per node? eg if I say dest=/foo/bar/$VAR-thing can I have $VAR change per node?17:57
clarkbgreghaynes: mordred if running in pull mode17:57
mordredclarkb: yes17:57
SpamapSgreghaynes: did you just invent that?17:58
greghaynesSpamapS: yes17:58
mordredclarkb: although I do not know anything about pull mode other than it seems to destroy the main awesome thing about ansible17:58
SpamapSgreghaynes: well done17:58
greghaynesSpamapS: naming it is the hard part17:58
SpamapSno the invalidation is17:58
mordredclarkb: however, I do not think that pull mode should break it eitehr way17:58
*** hdd has joined #openstack-infra17:58
clarkbmordred: d-g needs to copy logs from all of the subnodes onto the primary node for jenkins scp to work17:58
clarkbmordred: its possible we can optimize that away when we go completely to swift17:59
*** jamesmcarthur has quit IRC17:59
clarkb(but I like the main job script to not know how to deal with its logs so that its flexible)17:59
*** claudiub has quit IRC18:00
clarkbmordred: so what would that look like? iirc ansible does all of its variable stuff on the local side not the remote side? I suppose it still knows about the node18:00
openstackgerritSteve Martinelli proposed openstack/requirements: Add heat-translator to global requirments projects
clarkbmordred: is the todo I am trying to fix18:01
mordredclarkb: it depends - but you're using a few words in a way that I'm goign to have to ask annoying questions to make sure we're talking abot the same thing18:01
mordredclarkb: AH! excellent. let me look at that18:01
pleia2can I get an infra-core quickly look at ? it's old, simple, and the author abandoned it last week (gave up?), but it's a good doc fix that we need so I restored18:02
*** gordc_afk is now known as gordc18:02
mordredclarkb: ok. so what you were thinking was a thing to run on each remote node that would scp the files back to the master node, yeah?18:03
mordredclarkb: rather tahn a thing that ran on the master node and scp'd from each node18:03
*** jogo has quit IRC18:03
clarkbmordred: no, I want to use ansible synchronize (which is rsync) to do it in parallel18:03
openstackgerritMerged openstack-infra/elastic-recheck: Make the query for bug 1403284 more generic
openstackbug 1403284 in OpenStack-Gate "mysql (1040, 'Too many connections')" [Undecided,Fix committed] - Assigned to Joe Gordon (jogo)18:04
clarkbmordred: I could also have nasible run asnible directly in parallel but thats kinda silly since syncrhonize works18:04
clarkbmordred: er ansible run rsync directly18:04
mordredclarkb: sure ... one sec18:04
pelix@zaro yeap something like jenkins-api-mock but could opt for just a simple HTTPServer that handles requests and you indicate either based on the present of request headers what the response should be, or a Queue of responses to be served irrespective of the request.18:04
*** dhritishikhar_ has quit IRC18:04
*** dhritishikhar_ has joined #openstack-infra18:05
clarkbmordred: then each node should go into a known subdr18:05
mordredclarkb: yeah. - that part is the easy part18:05
*** EmilienM|afk is now known as EmilienM18:06
dhellmannclarkb, anteaya: pinging again on -- problems with that job are blocking us from merging an unrelated change in oslo.messaging that we need for the stable/kilo release18:08
pelixzaro: indeed --workers probably not needed test at all unless the write out to files is put inside the function called in parallel18:08
openstackgerritLevi Blackstone proposed openstack-infra/project-config: Enable gate jobs for remaining StackTach libraries
*** davideagnello has quit IRC18:10
mordredclarkb: ok - so I undersatnd what you want18:10
clarkbdhellmann: doesn't that need swift enabled for glance?18:10
*** jtriley_ has quit IRC18:11
dhellmannclarkb: good question. jogo put the patch together based on an earlier patch for the same bug18:11
dhellmannand now I don't see jogo on irc to ask18:11
clarkbsdague may know18:11
clarkbsdague: on is my -1 valid?18:12
clarkbI am pretty sure it is18:12
clarkbthe only thing that would make it invalid is devstack magic aiui18:12
*** jamesmcarthur has joined #openstack-infra18:12
*** davideagnello has joined #openstack-infra18:12
dhellmannclarkb: I don't see swift enabled in any other place where that same line is in the same file18:13
*** yamahata has quit IRC18:14
clarkboh the other large ops job does this too18:14
clarkbI am probably wrong then I can change my vote18:14
clarkb(that seems odd that we test glance differently here)18:14
dhellmannyeah, it does, but maybe that's part of the way we collapsed some alternate case jobs together?18:15
mordredclarkb: OH! I knew there was somewhere we were missing each other18:16
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Added parallelization options
mordredclarkb: when you said "pull mode"- you meant "synchronize's pull mode" - I thought you meant ansible pull mode18:17
mordredclarkb: which is why I was confused18:17
mordredclarkb: one sec - it's easy18:17
clarkbmordred: yes synchronize pull mode. I have no idea what ansible pull mode is18:17
mordredclarkb: you never should18:17
mordredclarkb: it's a dumb idea18:17
mordredclarkb: (it's the thing that turns ansible into puppet agent)18:17
*** dboik_ has quit IRC18:17
dhellmannclarkb: ty18:17
*** dboik has joined #openstack-infra18:18
clarkbmordred: that sounds amazing (I am guessing that its easier for network security in some orgs to have a  bunch of things connect to one place nevermind that its spofy and scales poorly)18:20
*** Sukhdev has quit IRC18:20
pleia2clarkb: thanks18:20
clarkbdhellmann: pleia2 np18:20
openstackgerritAlexander Tivelkov proposed openstack-infra/project-config: Added branch creation to yaql ACL
*** Hal has joined #openstack-infra18:20
*** pelix has quit IRC18:20
*** jcoufal has quit IRC18:20
*** Hal is now known as Guest6162818:21
mordredclarkb: I tested a similar thing locally against a remote host and copied things back18:21
mordredclarkb: but consider that untested in terms of $BASE expansion, etc18:21
openstackgerritMerged openstack-infra/nodepool: Update Readme config -> system-config repo
clarkbmordred: except we want a known constant dir18:21
mordredclarkb: sure18:22
clarkbmordred: because it makes other tooling that looks for logs simpler18:22
mordredclarkb: you explicitly want the numbered subdir, rather than one based on hostname, yeah?18:22
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack
clarkbmordred: ya (or some other arbitrary but constant set of iterations)18:22
clarkbmordred: so a b c d etc is fine too18:23
clarkbmordred: but $random-nodename or $random-ip is not so good18:23
*** tiswanso_ has quit IRC18:23
openstackgerritMerged openstack-infra/project-config: Reduce number of services running in largeops-src
mordredclarkb: so we should, for this iteration, put a counter into the inventory18:23
mordredwith a variable name18:23
mordredfor each host18:23
clarkbmordred: ah! ok I can do that18:23
mordredand refernece that instead of inventory_hostname18:23
*** tiswanso has joined #openstack-infra18:23
clarkbmordred: thanks this helps18:23
mordredclarkb: woot!18:23
* mordred was helpful today18:23
*** Guest61628 has quit IRC18:23
*** tnovacik has joined #openstack-infra18:24
*** AJaeger_ has joined #openstack-infra18:25
AJaeger_pleia2: reading the scrollback: Should we add you as admin to the openstack organization in transifex so that you have full access?18:25
*** unicell has quit IRC18:26
*** unicell1 has joined #openstack-infra18:26
clarkbmordred: I think this will help with the network setup too since I can use the same counter to set IPs18:26
mordredclarkb: ++18:27
clarkbmordred: do I have to do anything specail to refer to an inventory var or would it just be {{ host_counter }} if host_counter=X is what I put in the inventory?18:28
pleia2AJaeger_: that would probably be helpful, I won't touch anything :)18:29
mordredclarkb: just host_counter=X18:29
mordredclarkb: and then {{ host_counter }}18:29
clarkbmordred: so inventory vars are "global" for each node?18:30
mordredclarkb: yes18:31
mordredclarkb: you can also do complexer variable data structures and namespacing - but then you're talking multiple files and stuff18:31
*** rkukura has quit IRC18:33
*** zz_zz_zz_ja is now known as ja18:33
*** signed8bit has joined #openstack-infra18:33
*** tonytan4ever has joined #openstack-infra18:33
AJaeger_pleia2: what'y your login?18:34
pleia2AJaeger_: good question! checking18:34
otherwiseguyanteaya: I assume infra uses a PyPI mirror for the CI stuff? I just added the first release of the 'ovs' python package to PyPI. When does that mirror update (if it exists)?18:34
AJaeger_clarkb, mordred, do you agree with adding pleia2 as transifex admin?18:34
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Add host counter to ansible inventory
clarkbAJaeger_: ++ especially since she is doing a lot of the zanata work18:34
mordredAJaeger_: ++18:35
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Add host counter to ansible inventory
mordredotherwiseguy: the mirror runs every ... I can't remember, 5 minutes?18:35
clarkbstill no jogo but he will likely be interested in ^ too18:35
clarkbmordred: ya 5 minutes18:36
AJaeger_clarkb, mordred: Thanks for the quick answer. I'll add her once she found her login ;)18:36
pleia2AJaeger_: pleia218:36
fungiotherwiseguy: normally within about 5 minutes, but at the moment i'm having to refresh the mirrors so it could still be a couple hours until that completes18:36
AJaeger_pleia2: that was too easy ;)18:36
otherwiseguymordred: ah, so we were just *really* fast with adding the networking-ovn/global-requirements patches. :)18:36
clarkbmordred: otherwiseguy but fungi kicked off some full syncs to catch up after pypi glusterfs troubles18:36
otherwiseguyfungi: ah. ok. just making sure it wasn't a completely manual process I needed to poke someone about. :) Thanks!18:37
*** _nadya_ has quit IRC18:37
*** [HeOS] has quit IRC18:38
AJaeger_pleia2: you should be setup now18:39
fungibtw, still no response from fantastical support, so i've got downloads of the latest setuptools wheel from pypi.dfw going in a continuous loop and validating the checksums. as soon as it breaks on a failure i'll see if i can get some more details from our logs and a tcpdump18:39
*** teran has quit IRC18:39
pleia2AJaeger_: thanks18:40
*** teran has joined #openstack-infra18:40
AJaeger_pleia2: the admin UI is not always intuitive, you might get lost - like I'm currently trying to find a page that shows that you're an admin ;)18:41
AJaeger_pleia2: includes you...18:41
*** ildikov has joined #openstack-infra18:42
AJaeger_but you're not seen here: - strange ;(18:42
AJaeger_pleia2: now you're added, forgot to save ;(18:43
AJaeger_pleia2: if you have questions, feel free to email or catch me on IRC...18:43
pleia2AJaeger_: will do, thanks for your help18:43
*** teran has quit IRC18:44
AJaeger_cinerama: btw. to get the list of projects with translations, you can either check or check in project-config the file zuul/layout.yaml18:44
*** ddieterly has quit IRC18:44
cineramaAJaeger_: cool, thanks18:45
AJaeger_repos that are translated have translation-jobs or openstack-doc-translation added18:45
AJaeger_and adding a new project to transifex means additing it in project-config, additing it in transifex - after creation of the initial file po file in the repo itself18:46
*** e0ne has joined #openstack-infra18:46
*** ddieterl_ has joined #openstack-infra18:46
AJaeger_Our tools only work if there's an existing po file, can be empty for a start and then we import we the toolchain18:46
pleia2oh yes, that's a clever way of finding them in configs18:47
sdagueclarkb: - the -1 was invalid. Devstack sets up glance to use swift if it's enabled, otherwise it works with a file system backend.18:47
*** Longgeek has quit IRC18:48
*** jamesmcarthur has quit IRC18:48
*** unicell has joined #openstack-infra18:48
*** unicell1 has quit IRC18:49
zaroclarkb: do you know which version of virtualenv jenkins machine uses when running 'tox -epy27'?18:50
*** hyakuhei has quit IRC18:50
*** AJaeger_ has quit IRC18:52
*** andrey-mp has joined #openstack-infra18:53
*** rkukura has joined #openstack-infra18:54
pleia2not sure how to respond to this, we kind of make the rules here, particularly as bots leave our controlled #openstack-* namespace
pleia2(and we gate on this, so it can't land)18:54
*** Rockyg has joined #openstack-infra18:55
*** marun_ has joined #openstack-infra18:55
*** rbradfor has quit IRC18:56
*** tonytan4ever has quit IRC18:56
*** bswartz has joined #openstack-infra18:57
*** Sukhdev has joined #openstack-infra18:58
*** marun has quit IRC18:58
andrey-mphi! who can help me with rally experimental job for project stackforge/ec2api? It worked two weeks ago but not now...  it fails becuase flag ERROR_ON_CLONE now is True in this job. I don't understand when/where it has changed last two weeks. And I don't understand how to fix it.19:00
*** baoli has quit IRC19:01
*** dkranz has quit IRC19:01
*** pabelanger has quit IRC19:01
anteayaotherwiseguy: I was out for a walk, looks like you got your response, which is better than I could have given you anyway19:01
otherwiseguyanteaya: thanks. :)19:02
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack
pleia2thanks fungi19:03
*** dizquierdo has quit IRC19:04
fungipleia2: also your answer was great. running a separate gerritbot should be trivial for them (they can even set up a separate account for it to authenticate against if they want)19:04
*** davideagnello has quit IRC19:04
*** sputnik13 has quit IRC19:04
* pleia2 nods19:05
pleia2I had a look at the readme to confirm first, it's easy stuff19:05
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral
*** sputnik13 has joined #openstack-infra19:06
*** davideagnello has joined #openstack-infra19:06
fungiyep, it is. apt-get install gerritbot, tweak config, copy initscript from our puppet module if you want, and you're off to the races19:06
fungier, not apt-get install but pip install19:06
*** kgiusti has joined #openstack-infra19:06
*** pabelanger has joined #openstack-infra19:07
fungihrm, we tagged 0.2.0 a year ago but it never landed on pypi19:07
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral
fungiahh, it has no pypi jobs!19:08
jklareAJaeger: ping19:08
*** jog0 has joined #openstack-infra19:09
fungino, nevermind. misread.  it does now, but maybe it lacked them when 0.2.0 was tagged a year ago19:09
sdaguedid we have *another* gerrit restart?19:09
*** AL34N1X has joined #openstack-infra19:09
sdagueoh, never mind, stale gerrit page19:09
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Rename stackforge/mistral to openstack/mistral
*** dprince has joined #openstack-infra19:11
fungilooking at the clustering of pypi.dfw read timeouts in logstash, it seems to come in spurts19:11
*** dguerri is now known as _dguerri19:11
*** Somay has joined #openstack-infra19:12
fungiwe had none from ~16:40-18:30 but then 7 within a second of one another19:12
clarkbsdague right amd we test swift + glance everywhere but here19:12
*** tonytan4ever has joined #openstack-infra19:12
clarkbis that intentional?19:12
sdagueyeh, there is nothing useful that swift does in largeops19:13
sdagueexcept take up memory19:13
sdaguethe hypervisor is fake19:13
*** hyakuhei has joined #openstack-infra19:13
*** hyakuhei has quit IRC19:14
*** spredzy_|afk is now known as spredzy_19:16
openstackgerritJan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific
jeblairfungi: i strongly suspect i will not be able to attend infra meeting tomorrow19:16
fungijeblair: no worries. i'll be happy to chair. anything you need added to the agenda?19:17
jklarecould someone help me a bit and tell me if this patch works as decribed in the commit (especially the pacemaker part)
*** _dguerri is now known as dguerri19:17
fungijeblair: i'll also lurk tc and cross-project as usual tomorrow in case anything comes up19:17
openstackgerritMerged openstack-infra/project-config: networking-nec: add ability to create and manage branches
jeblairfungi: don't think so -- remind folks about the summit prep etherpad?  i think i'll collate that end of week/early next week.19:18
jeblairfungi: cool, thanks19:18
fungijeblair: yep, will definitely do19:18
jeblairttx: i'll probably miss the tc meeting too; will vote on tc changes today.19:18
*** jtriley has joined #openstack-infra19:19
*** Somay has quit IRC19:19
*** shashankhegde has quit IRC19:19
andrey-mpcould this patch break behavior of ERROR_ON_CLONE for this ?19:19
*** jcoufal has joined #openstack-infra19:21
*** yamahata has joined #openstack-infra19:21
*** Somay has joined #openstack-infra19:22
*** sdake_ has joined #openstack-infra19:22
openstackgerritJulia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through
zarofungi, clarkb : do you know which version of virtualenv jenkins machine uses when running 'tox -epy27'?  i don't see it in the logs.19:23
*** jog0 is now known as jogo19:23
clarkbzaro should be whatever latest tox includes19:23
pleia2off to lunch + weekday errands, bbiab19:24
clarkbya I'm mostly not here atm19:24
zarohmm, then i'm stumped as to why zuul test only fail when run with tox.19:25
*** AL34N1X has quit IRC19:25
*** sdake has quit IRC19:26
*** annegentle has joined #openstack-infra19:27
greghaynesmordred: interesting dib regression for youall (I think) - we are going to be using default apt/yum mirror info now where as before the rax/hp mirror info was baked in to be a localish mirror19:28
*** otter768 has joined #openstack-infra19:28
greghaynesthis is currently the case in hpcloud, obviously19:28
*** rbradfor has joined #openstack-infra19:28
jeblairgreghaynes: i think that's a desired feature, actually :)19:29
*** armax has quit IRC19:29
*** ociuhandu has quit IRC19:29
jeblairgreghaynes: when phrased as "do not use broken local mirror..." :)19:29
greghayneshah, ok then. The downside being default mirrors tend to be slow19:30
greghaynesbut as long as were aware19:30
jeblairgreghaynes: yeah, we might want to run our own mirrors.  we talk about that occasionally.19:30
*** armax has joined #openstack-infra19:31
*** MaxV has joined #openstack-infra19:31
openstackgerritlifeless proposed openstack-dev/pbr: Use /opt/git directly
jeblairgreghaynes: but also, hopefully we will use the mirrors very little -- on-node cache should help a lot19:32
openstackgerritlifeless proposed openstack-dev/pbr: Stop testing install
openstackgerritlifeless proposed openstack-dev/pbr: Test pip install -e of projects.
openstackgerritlifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many.
openstackgerritlifeless proposed openstack-dev/pbr: Improve
greghaynesjeblair: ah, good point19:32
*** dguerri is now known as _dguerri19:32
andrey-mpfungi, jeblair, sdague:  who can help me with rally experimental job for project stackforge/ec2api? It worked two weeks ago but not now...  it fails becuase flag ERROR_ON_CLONE now is True in this job. I don't understand when/where it has changed last two weeks. And I don't understand how to fix it. I think that this patch changed behavior -  https://rev19:32
andrey-mpor maybe I should file a bug for devstack?19:33
*** otter768 has quit IRC19:33
lifelesscould I beg two x +2 and a +A for <- it was green and will be again - that push was just to tweak a comment for jhesketh19:33
*** ja is now known as zz_ja19:33
*** hdd has quit IRC19:34
sdagueandrey-mp: I don't know, we never set that to true19:34
jeblairandrey-mp: i can't dig into it now, but in general, you should add stackforge/ec2api to the PROJECTS variable in the job19:34
mordredgreghaynes: also - AFS solves this ...19:34
*** Somay has quit IRC19:35
*** dkranz has joined #openstack-infra19:35
*** sdake_ has quit IRC19:35
sdagueandrey-mp: so, ERROR_ON_CLONE=True is true for all gate jobs19:35
sdagueyou are not allowed to do that differently19:35
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: Put input variables into ansible inventory
openstackgerritMonty Taylor proposed openstack-infra/devstack-gate: Move all the ansible calls into playbooks
*** sdake has joined #openstack-infra19:35
*** Somay has joined #openstack-infra19:35
mordredclarkb: I couldn't help myself - so I did that ^^ feel free to -2/abandon the if you hate it - no emotional attachment whatsoever19:35
andrey-mpsdague: yeah, i saw this in all files. but previous gating had a line '+ [[ False = \T\r\u\e ]]' and error didn't raise19:35
mordredclarkb: it was more a "I started poking and couldn't stop poking"19:36
greghaynesthe "AFS solves this"19:37
andrey-mpjeblair: ok. (ec2api already in PROJECTS)19:37
*** isviridov_away is now known as isviridov19:37
sdagueandrey-mp: this is why it doesn't work -
openstackgerritJan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific
*** isviridov is now known as isviridov_away19:38
andrey-mpsdague: thank you very much ) i will dig into this now19:39
mordredgreghaynes: (so, it's unlikely to be how we solve it - but if we had apt/yum mirrors in AFS, and we had an AFS fileserver per cloud region, then the config for the nodes could be a completely static apt source of file:///afs/ and the AFS infrastructure would direct the cache misses to the closest file server)19:39
fungisdague: andrey-mp: rally moved from stackforge to openstack namespace. that's likely what caused it to start failing?19:39
sdaguefungi: yes, that's what it looks like19:40
*** marun_ is now known as marun19:40
jeblairandrey-mp: change stackforge/rally to openstack/rally in that job defn19:40
jeblairoh yep. :)19:40
* jeblair goes back to laggy conf wifi19:40
mordredlifeless: I see you're back to teh while loop -- did we learn something?19:41
andrey-mpwow, thank you all very much ))19:41
* mordred wants to make sure the pip hole in his brain is appropriately filled19:41
lifelessmordred: no, its a stack19:41
mordredlifeless: okie19:41
mordredso I just haven't gotten back to the "remove the while loop" patch19:42
* jeblair wonders if internet3 conference wifi will be better19:42
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: fix ec2-api rally job
nibalizerjeblair: can you just get on wifi2 ?19:43
jeblairnibalizer: i should totally set up an ap with that ssid here19:44
*** hdd has joined #openstack-infra19:44
lifelessinternet3? I didn't think 2 was finished yet19:44
mordredlifeless: stack looks good19:44
*** ibiris is now known as ibiris_away19:45
mordredlifeless: all +2 from from me19:45
openstackgerritMichael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0
mordredclarkb: when you get a sec - could you relook at 176024 - you've got a -1 on it which SpamapS addresses in teh follow up patch19:49
*** hdd has quit IRC19:49
*** [HeOS] has joined #openstack-infra19:51
*** emagana has joined #openstack-infra19:52
*** ildikov has quit IRC19:53
*** dhritishikhar_ has quit IRC19:53
*** jamesmcarthur has joined #openstack-infra19:54
lifelessmordred: so - the q about failed wheels is moot now19:54
lifelessmordred: everything succeeds anyway19:55
lifelesstotal time 6 minutes19:55
*** jcoufal has quit IRC19:55
*** andrey-mp has quit IRC19:56
lifelessvs 11 minutes19:56
lifelessso its a nice improvement19:56
mordredlifeless: woot19:57
*** armax has quit IRC19:58
openstackgerritTristan Cacqueray proposed openstack-infra/puppet-openstackci: Create a jenkins server like
openstackgerritMichael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0
openstackgerritlifeless proposed openstack-dev/pbr: Stop testing easy_install behaviour
nibalizerso what state is in ? I can't tell a good reason why it hasn't landed19:58
nibalizeri even tried reverify which im pretty sure isnt a thing19:59
mordrednibalizer: I nudged it with a second +A19:59
mordredlet's see what happens19:59
fungibtw, i manually triggered the release (tarball/wheel, pypi uploads) jobs got the gerritbot 0.2.0 release from a year ago20:00
mordredI also +2'd your readme change - but is there a reason we have files?20:00
mordredfungi: neat20:01
mordredfungi: we wanted that, right?20:01
*** ildikov has joined #openstack-infra20:01
fungimordred: i assume so. we're running 0.2.0.post6 on review.o.o for a while now20:01
fungimight even be time for a 0.3.0 or something20:01
*** dhritishikhar_ has joined #openstack-infra20:02
fungiso the 3-year-old 0.1.2 sdist on pypi was sort of misleading20:02
*** marun has quit IRC20:03
*** dhritishikhar_ has quit IRC20:04
jesusaurusfungi: jeblair: clarkb: can you add 74943 to your to-review list?20:05
*** dhritishikhar_ has joined #openstack-infra20:05
*** _nadya_ has joined #openstack-infra20:05
mordredfungi: ++20:06
*** johnthetubaguy is now known as zz_johnthetubagu20:06
mordredfungi: one MIGHT even go so far as to say it might be time for a 1.020:06
fungiindeed it might at that20:07
fungijesusaurus: mordred: i take it our requests socket timeouts for provider api connections from nodepoold don't obviate the need for 74943?20:07
fungifor example,
*** doude has quit IRC20:09
*** doude has joined #openstack-infra20:09
mordredfungi: uhm20:09
fungioic, the concern is that if you timeout the api during an image update, it may be lost20:09
fungiso this is explicit to proxyerror exceptions then20:11
fungier, specific to20:11
*** Swami has quit IRC20:11
*** dhritishikhar_ has quit IRC20:11
mrmartinfungi: may I ask you to do a pip freeze on ask.o.o and paste the output to paste.o.o ?20:12
fungimrmartin: `pip list` output
*** jamesmcarthur has quit IRC20:14
*** teran has joined #openstack-infra20:14
lifelessmordred: so I'm wondering20:15
fungimordred: jesusaurus: got it, this is not solving a problem of the api endpoint ceasing to respond and the socket apparently going dead. we get an actual exception raised in this case... going back over it now with an eye to that as the problem statement20:15
lifelessmordred: is there enough memory to run against tmpfs?20:15
mordredfungi: yah - it's a different thing20:15
mordredlifeless: there is 8G of ram20:15
lifelessmordred: and then partition it per cpu ?20:15
lifelessso yes20:15
lifelessI'll get the devstack-gate process removed first20:16
*** dprince has quit IRC20:16
lifelessand then potter with that I think, as 90m test runs are uhm, poor.20:16
jesusaurusfungi: yeah, the ProxyError is orthogonal to the api timeout, the error may be raised immediately20:16
jesusaurusfungi: in which case we should create a new client object (which reconnects through the proxy) and try again20:17
lifelesswill have to see what its like with the easy_install stuff removed, which hopefully is the only truely uncached run20:18
*** _dguerri is now known as dguerri20:20
fungii do like that trick we're using in nodepool for raising an exception from a subthread in the main thread. that's nifty20:23
*** shashankhegde has joined #openstack-infra20:26
mrmartinfungi: what is the base image we are using for ask.o.o in rax?20:26
jesusaurusfungi: im glad you like that, it took me a while to realize that the original patch was raising the exception in the wrong thread20:29
*** MarkAtwood has quit IRC20:29
*** Rockyg has quit IRC20:29
fungimrmartin: Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM)20:29
*** yfried_ is now known as yfried|afk20:30
fungimrmartin: and flavor is "4 GB Performance"20:30
*** hdd has joined #openstack-infra20:31
mrmartinyeah, great, and is it a standard 12.04LTS cloudimg?20:31
fungiyep, and then we apply our puppet manifest for ask.openstack.org20:31
fungimrmartin: well, depends on what you mean by "standard 12.04LTS cloudimg"20:32
mrmartinthe puppet part is ok. I'm playing with ask.o.o upgrade, and it works well in my vagrant box based on 12.04lts, but when i try to deploy it into a HP Helion public cloud20:32
*** tiswanso has quit IRC20:32
fungimrmartin: it's whatever rackspace is calling "Ubuntu 12.04 LTS (Precise Pangolin) (PVHVM)" but it's not the image provided from for precise20:32
mrmartinthings start to broke, and it seems to be a lot of extra python packages included by default20:32
fungirackspace does things to make images work for their custom cloud environment20:32
mrmartinsomething like this:
*** hdd has quit IRC20:33
mrmartinso much-much more that we have in production ask.o.o20:33
fungimrmartin: hpcloud uses images from, which i think have cloud-init preinstalled, and it's python-based with a fairly lengthy set of python module dependencies20:34
mrmartinand one of the python packages are killing the new askbot deployment20:35
mrmartinI don't whether a custom virtualenv deployment could solve my issue20:35
mrmartinand deploy only those packages there that we required20:35
fungipossible you could uninstall cloud-init and its dependencies20:35
mrmartinI don't = I don't know20:35
mrmartinok, but if I uninstall cloud-init, what will happen after a reboot?20:36
*** dizquierdo has joined #openstack-infra20:36
lifelessdstufft: may interest you20:36
*** annegentle has quit IRC20:36
fungimrmartin: not sure, but we definitely run instances in hpcloud without cloud-init installed20:37
*** mwagner_lap has quit IRC20:37
mrmartinok, I'll play with that tomorrow, and try to build up a similar image20:37
openstackgerritMerged openstack-infra/nodepool: Reset the client object after proxy timeouts
fungimrmartin: i take that back, we seem to have cloud-init installed even on our dib-built images in hpcloud, but we may be installing it via pip instead of debs20:38
*** dguerri is now known as _dguerri20:39
*** _dguerri is now known as dguerri20:39
*** whoops has joined #openstack-infra20:39
*** _nadya_ has quit IRC20:39
*** mrunge has quit IRC20:42
jogoclarkb fungi: not sure if that is a one off or not20:43
*** miqui has quit IRC20:43
fungijogo: ssh connectivity failure to a subnode?20:44
jogofungi: yup20:44
*** eharney has quit IRC20:44
fungifollowing the logstash link from elastic-recheck for the pypi.dfw tcp read timeouts, i see some rally jobs which look like they're not being indexed into logstash correctly. anybody know what the cause is there? line splitting seems to be broken20:46
fungimaybe the console log format for those doesn't match similar jobs20:46
mrmartinfungi: if the infra approve this ask-staging patch: tomorrow, what is the earliest date when we can use that?20:46
*** purp has quit IRC20:47
*** purp has joined #openstack-infra20:47
*** tqtran has quit IRC20:48
*** MaxV has quit IRC20:48
fungimrmartin: i'll probably be fairly busy since meetings tend to take up most of my tuesdays, but i should have time to try launching the new server for it on wednesday20:48
mrmartinok, great I try to bring the patch into a proper form20:48
fungimrmartin: thanks--clarkb makes some good points in his comments20:49
mrmartinso it'll deploy a ask application environment where we can play20:49
mrmartinyeah I see that20:49
fungiand i guess we want it deploying from the feature/development branch20:49
jklareis there currently a known issue with the gates?20:49
*** yfried|afk is now known as yfried_20:50
jklaretimed out two times in a row at different points20:50
mrmartinso clarkb do you suggest to rename the staging site to ask-dev.o.o or rename the manifest to ask-staging ?20:50
fungijklare: that second one doesn't look like a timeout. looks like a git fetch error20:51
*** turul has joined #openstack-infra20:51
fungijklare: in a different job20:51
*** turul is now known as afazekas20:52
jklarefungi: yeah, i thought it might be due to a connection issue (which lead to a timeout in the first case)20:52
fungijklare: and i can repeat that fetch successfully now, so chances are there was some intermittent network issue between that worker in hpcloud and the zm07 zuul merger in rackspace20:53
*** MaxV has joined #openstack-infra20:53
jklareso just recheck?20:53
fungijklare: the timeout in the previous check run on the puppet apply job for centos6 is a known intermittent issue we've been unable to reliably reproduce to be able to troubleshoot further20:53
*** shardy is now known as shardy_z20:53
fungijklare: yes, looks like rechecking should be fine20:53
*** hichihara has joined #openstack-infra20:54
fungias for centos6, apparently we sometimes see git remote calls fail just on that platform. and the puppet apply job tries to update some 50+ git repos, so it's exposed there more often20:54
jklaremmh... kk20:55
*** yamahata has quit IRC20:55
fungibut they fail in such a way that the socket is open blocked on read and eventually the job times out at 30 minutes20:55
*** yamahata has joined #openstack-infra20:55
fungii've tried desperately to reproduce teh condition manually, to no avail20:56
fungiat this rate, we'll drop centos 6 as a testing platform before we identify the issue20:56
jklaresounds reasonable20:56
openstackgerritSpencer Krum proposed openstack-infra/system-config: Convert hiera configuration to support public data
*** Sukhdev has quit IRC20:58
*** bswartz has quit IRC20:59
jklarefungi: thanks for the fast response20:59
*** yfried_ is now known as yfried|afk21:00
*** tkelsey has quit IRC21:00
sdakedo infra have sesions at smmit?21:00
sdakethe kolla upstream would like to have a session on adding a docker registry to the openstack infrastructure if possible21:00
*** amotoki has joined #openstack-infra21:01
*** MaxV has quit IRC21:01
sdakeis there an etherpad for it spamaps?21:01
sdakethanks fungi21:01
SpamapSsdake: a whole session might be a bit much.. but I'd bring that up now, and get the discussion going now, so we can just hammer out face to face details. :)21:01
*** Sukhdev has joined #openstack-infra21:01
mordreddocker registry should not be a hard task21:02
SpamapSSeems like a pretty straight forward thing to run though.21:02
SpamapSand might help with things like running devstack in docker21:02
*** armax has joined #openstack-infra21:03
sdakeSpamapS right - I have no idea what it would take21:04
sdakefungi thanks for  the link, I added the topic there21:04
sdakewe can just discuss face to face if that is the infra cores preference21:04
sdakebut right now a docker push inside our gate would take about 5-8 hours, so we can't do that with the 1 hour timelimit.21:04
sdakea local registry would be much faster - 30 minutes or so21:04
mordredsdake: don't forget that we have nodes that span clouds21:05
*** marun has joined #openstack-infra21:05
mordredso a push may not be local21:05
sdakemordred good point21:05
mordredbut I still think it's a useful thing to put on the list21:06
sdakethanks - its on the list21:06
clarkbok mostly back now, catching upafter long lunch then finiahing capstone prep stuff21:07
*** annegentle has joined #openstack-infra21:07
*** sdake_ has joined #openstack-infra21:07
*** annegent_ has joined #openstack-infra21:07
*** annegentle is now known as Guest9221421:07
mordredclarkb: I made you a couple of patches21:08
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Add dstat to subnode in multinode mode
jogoclarkb: ^21:09
*** stevemar has quit IRC21:10
*** sdake has quit IRC21:11
*** Swami has joined #openstack-infra21:11
*** esker has quit IRC21:12
*** rfolco has quit IRC21:12
clarkbjogo: commented21:13
clarkbmordred: they fail all the tests :P21:14
*** ldnunes has quit IRC21:14
*** jogo has quit IRC21:15
EmilienMrubygems server is *too* often down, which slow down Puppet CI (we download gems from the official server directly). Would os-infra agree to host a mirror ( the tools are already in place : )21:15
EmilienMnibalizer: ^21:15
*** sdake has joined #openstack-infra21:15
EmilienMwe can pay with beers.21:16
*** esker has joined #openstack-infra21:18
clarkbany opposition to me self approving 173603?21:19
clarkbmordred: ^ you may be intersted in the set of slides there21:19
clarkbEmilienM: anyone can propose the change to add the server21:19
*** sdake_ has quit IRC21:19
*** Guest92214 has quit IRC21:19
clarkbEmilienM: it would probably be best if someone familiar with gems and its mirror mechanism did that21:20
*** tonytan4ever has quit IRC21:20
*** dustins has quit IRC21:20
*** gyee has quit IRC21:20
EmilienMclarkb: will look21:22
nibalizerclarkb: so are you thinking one mirror, or one mirror per az? or jsut colocating gem mirrors on the pypi mirrors ?21:22
nibalizerEmilienM: im happy to help you get the rubygems mirrors up if -infra is cool with the overall idea21:23
clarkbnibalizer: probably colocating on the pypi mirrors unless the dfw pypi mirror is derping due to load21:23
jeblairEmilienM, nibalizer: ++21:24
jeblairclarkb: ++; should probably rename those "mirror." and shift things down a path level.  but we can do that later.21:24
jeblair(and dust off afs)21:25
*** tjones1 has quit IRC21:25
jeblair(so we only have to run the mirror jobs in once place)21:25
greghaynesSpeaking of mirrors, I have some patches to add wheels to the mirrors that coule use review,n,z21:25
clarkbmordred: SpamapS re 176024 there is still the issue in the tests21:26
clarkbmordred: SpamapS can we confirm its enforcing bug behavior before ew let that slip in?21:26
clarkbmordred: SpamapS it looks to me more like Mock assert truee because thats what it do21:26
clarkbthouh on closer reading args is not what is passed to the cretion it is used to assert what creation was clled with21:29
*** otter768 has joined #openstack-infra21:29
clarkbso I may be wrong about the mock being wrong21:29
lifelessttx: mordred:
*** mrmartin has quit IRC21:33
*** otter768 has quit IRC21:34
*** dkranz has quit IRC21:34
*** ZZelle_ has joined #openstack-infra21:35
sigmavirus24lifeless: you live in the future21:37
lifelesssigmavirus24: arguably :)21:38
*** jgrimm is now known as zz_jgrimm21:38
*** vhoward has quit IRC21:41
openstackgerritSteven Dake proposed openstack-infra/project-config: Let kolla-release create branches
*** vhoward has joined #openstack-infra21:42
*** mriedem has quit IRC21:43
*** sdake_ has joined #openstack-infra21:46
*** jtriley has quit IRC21:46
openstackgerritDavide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP support
clarkblifeless: I may have missed it but I don't think you address how proposal.3 and proposal.4 are supposed to help goals.3 and goals.4. basically what do distros do if we only every assert one set of things that no distro will ever support together21:46
fungiokay, i've figured out why bandersnatch wanted to redo its refresh after my manual refresh completed. it's not creating a /srv/static/mirror/status file, though it does create .lock and generation just fine21:47
clarkbgranted its not like we actually test the full spread of deps today anyways, but ew do seem to at least consider it a bug if say debian and fedora can't both do something reasonable with say python-libvirt21:47
*** isviridov_away is now known as isviridov21:47
*** sdake has quit IRC21:49
fungialso there's a fairly lengthy /srv/static/mirror/todo file21:49
fungiafter the resync completes21:49
*** armax has quit IRC21:50
*** sarob has quit IRC21:52
clarkbfungi: file permissions problem?21:52
clarkbfungi: or some other issue writing the status? maybe we ran out of disk?21:52
fungiclarkb: doesn't look like it. we run it as root and the files are owned by root21:52
fungiplenty of disk21:52
*** sarob has joined #openstack-infra21:52
greghaynesclarkb: SpamapS replied on, Im not sure its really a bug21:53
fungiclarkb: look on pypi.ord for an example (root screen session is still running there, untouched from when the manual resync completed)21:54
lifelessclarkb: the distros either don't have this problem (nixos), use containers to avoid it (juju), or don't support using things during upgrades21:54
fungiunfortunately before i noticed the missing status file on pypi.dfw i tried rerunning bandersnatch just to make sure it updated successfully... and so it's redoing a complete refresh again21:55
*** hasharAway- has quit IRC21:55
clarkblifeless: the last case assumes we are the only python thing asserting deps on the system21:55
clarkblifeless: in the case of red hat that is really not true (yum)21:55
*** marun has quit IRC21:55
lifelessclarkb: in the context of the gate and devstack ?21:56
clarkblifeless: no in the context of our users21:56
*** gordc has quit IRC21:56
lifelessmaybe I don't understand your statement21:56
lifelessyou're saying that redhat ignore our list of what works together and do their own thing?21:56
clarkblifeless: people want to use our software, it makes sense that they would like to deploy the asserted list of what works21:56
clarkblifeless: yes what other choice do they have?21:57
clarkbif we hvae pinned 300 python deps and half of them don't match what rhel ships21:57
lifelessclarkb: they can contribute a third party CI that tests with their shipping deps21:57
lifelessclarkb: and see community buyin on working on that set of deps21:57
lifelessclarkb: or they can seek community consensus that that should be the default test and shift the burden on other sets off to other people21:58
pleia2cinerama: camunoz of redhat/zanata will be at the summit, so I'll be sure to send along our feature requests/bugs prior to so I can have some productive sit down time in person :)21:58
*** isviridov is now known as isviridov_away21:58
lifelessclarkb: smokestack was basically that, no?21:58
clarkblifeless: ok so we are basically telling them its their thing to solve21:58
lifeless[the former choice]21:58
clarkblifeless: no smokestack was test on fedora with xen21:58
cineramapleia2: that's a great idea. i think we can work around some of the more critical things for now but it would be great to get some of those fixes21:58
lifelessclarkb: but also packaged deps21:59
lifelessclarkb: I remember danprince being quite specific about that catching issues21:59
clarkblifeless: we for the most part did mostly packaged deps until recently21:59
pleia2cinerama: indeed, I think I owe this project some theming research too, need to sift through bugs and things to see where we're at21:59
lifelessclarkb: so I'm not saying its 'their thing to solve'21:59
lifelessclarkb: I had a whole algebra for discussing this that I ripped out because ETOOLONG22:00
*** arielb has quit IRC22:00
clarkb(it was this cycle that devstack started pushing more things to pypi packages and not system packages)22:00
lifelessclarkb: in short though, their packaged versions are another constraint list.22:00
lifelessWe can run any number of constraint lists in parallel with the same infrastructure22:00
lifelessone set of jobs per list22:00
lifelesstesting known good is important for CI stability and us getting work done. Testing latest is important to keep us from ending up stale and stuck on unsupported or insecure versions.22:01
lifelessTesting whats in $distribution is important if we're going to change our code to work with that.22:01
lifelessI mean - if the consensus is 'we should work on whats out there today', then we should accept such bug reports22:02
lifelessand testing with such a constraint set would be a good way to avoid them ever becoming bugs.22:02
lifelessOTOH if the consensus is to require distros to upgrade to reasonable versions, then such bugs would be rejected, etc.22:03
clarkblifeless: well its less about upgrading necessarily and mor eabout we are asserting one version22:03
lifelessI think technically its better to work with as old versions as can be done with moderate effort.22:03
clarkblifeless: that may mean downgrades for some distros and upgrades for others22:03
*** bknudson has quit IRC22:03
lifelessDon't kill ourselves to work with old broken.22:03
lifelessBut don't force upgrades spuriously.22:03
clarkbbut it will most definitely not match anything that is out there22:04
lifelessclarkb: so *we* assert version X works. Redistributors have a choice: a) work with the version we tested. b) run tests themselves with a different version.22:04
*** yamamoto has quit IRC22:04
lifelessclarkb: Adding more upstream version tests (e.g. a set of 'lowest supported versions' as a parallel job)22:04
lifelessclarkb: would be one way of sharing the effort of b across the community22:04
*** e0ne has quit IRC22:05
*** kgiusti has left #openstack-infra22:06
*** amitgandhinz has quit IRC22:07
clarkbhrm jogo keeps disappearing, the failur ehe linked earlier is really weird22:07
clarkbssh was working then stopped, so something funky happened22:08
clarkb(rax network issue? VM stopped ssh daemon? I dunno)22:08
*** vhoward has quit IRC22:09
*** salv-orlando has quit IRC22:09
*** peristeri has quit IRC22:09
*** vhoward has joined #openstack-infra22:10
fungiapparently if i run bandersnatch mirror instead of using the run-bandersnatch wrapper, it picks up with the todo file where the initial refresh left off. i sort of patched dfw back into shape that i can continue it that way22:10
clarkbfungi: interesting, maybe we broke compat with the wrapper script then?22:11
fungihopefully this creates a proper status file at the end22:11
*** jamesmcarthur has joined #openstack-infra22:11
clarkblifeless: in general though looks good. We definitely have to fix pip before we can consider much of this22:12
clarkblifeless: so good to see you diving in there22:12
clarkblifeless: any idea if "fixing" pip will lead to backward incompatible installs for pypi ackages and is that a problem?22:12
jesusaurusthe log retention in logstash is 2 week, yes? how much disk space does that use up?22:12
clarkbjesusaurus: 10 days, looking at cacti now to get numbers22:13
clarkbhrm doesn't seem to be in cacti22:14
clarkbthats odd22:14
openstackgerritJulia Kreger proposed openstack-infra/shade: Enhance error message in update_patch
openstackgerritJulia Kreger proposed openstack-infra/shade: Update recent Ironic exceptions
openstackgerritJulia Kreger proposed openstack-infra/shade: WIP: node_set_provision_state to task
fungiclarkb: does cloning puppet modules into /etc/puppet/environments/clarkb/modules on the puppetmaster work for you with your current environment.conf there?22:14
clarkbjesusaurus: looks like ~260GB per node and there are 6 nodes. We can probably bump back to 2 weeks based on those numbers22:15
clarkbfungi: yes, but you have to make sure you clone them to the right name, eg puppet-foo cloned to foo22:15
fungiclarkb: perfect--thanks!22:15
clarkbfungi: feel free to use my env too if you need to22:15
clarkbfungi: I am not needing it for anything22:15
jesusaurusclarkb: thanks22:16
fungiclarkb: nah, i think this ought to get me on the right track. going to take a stab at serving a custom gerrit module from my env that doesn't rerun the lucene reindex when replacing a warfile, so we can retest the patched 2.8 war without excessive downtime22:17
*** vishy has quit IRC22:17
*** cyeoh has quit IRC22:17
lifelessclarkb: can I have a +A on
lifelessclarkb: bug 2687 is the one I expect to make us fall over in a heap22:19
openstackbug 2687 in scummvm (Ubuntu) "ScummVM depends on libflac6 which is not avalaible with Breezy" [Medium,Fix released] - Assigned to MOTU (motu)22:19
lifelessclarkb: we'll need to add --ignore-installed while we fix stuff up22:19
*** cyeoh has joined #openstack-infra22:19
*** vishy has joined #openstack-infra22:20
lifelessor fungi:
clarkblifeless: why did the old wheelhouse stuff not work?22:22
*** dizquierdo has quit IRC22:23
clarkblifeless: is PIP_FIND_LINKS not valid and pip ignores the env var?22:23
*** tnovacik has quit IRC22:23
*** marun has joined #openstack-infra22:23
clarkbpretty sure devstack has similar problems i so22:23
*** esker has quit IRC22:23
clarkblifeless: also should pipflags apply to the wheelhouse build commands?22:24
lifelessclarkb: the only hits for that variable in the pip codebase are in tests.22:24
*** mriedem has joined #openstack-infra22:25
clarkblifeless: pip is supposed to treat all command line flags as valid env vars too22:25
clarkbmagic around argparse maybe?22:25
lifelesslet me cross check22:25
*** amotoki has quit IRC22:27
lifelessclarkb: ah, I see - found it. Yes it does, so it probably worked22:27
lifelessclarkb: OTOH I think this is clearer22:28
*** Somay has quit IRC22:28
lifelessclarkb: because you don't need as much context to understand whats going on22:28
*** salv-orlando has joined #openstack-infra22:29
clarkblifeless: PIP_FIND_LINKS is much easier to understand than pip -f22:29
lifelessclarkb: only if you know its exported in the script22:30
clarkblifeless: and it allows anyone to add pip commands without needing to know the magic sauce22:30
clarkblifeless: which is already an issu because pipargs isn't used on some pip commands22:30
clarkbsorry PIPFLAGS22:30
lifelessknowing that there might be environment variables affecting things *is* magic sauce22:31
clarkbyes but its not necessarily knowledge22:31
lifelesswell it is - because I misunderstood the script because I didn't know that thing22:32
clarkbanyways left comment about PIPFLAGS22:33
*** bswartz has joined #openstack-infra22:33
*** mriedem has quit IRC22:33
*** sigmavirus24 is now known as sigmavirus24_awa22:33
*** mriedem has joined #openstack-infra22:34
*** nelsnelson has quit IRC22:34
*** tjones1 has joined #openstack-infra22:35
*** rbradfor has quit IRC22:35
*** dboik_ has joined #openstack-infra22:36
*** ZZelle_ has quit IRC22:37
*** camunoz_gone is now known as camunoz22:38
*** dangers is now known as dangers_away22:38
openstackgerritlifeless proposed openstack-dev/pbr: Use /opt/git directly
openstackgerritlifeless proposed openstack-dev/pbr: Stop testing easy_install behaviour
openstackgerritlifeless proposed openstack-dev/pbr: Test pip install -e of projects.
openstackgerritlifeless proposed openstack-dev/pbr: Build all the wheels in one pass rather than many.
*** mriedem has quit IRC22:38
openstackgerritlifeless proposed openstack-dev/pbr: Improve
lifelessclarkb: mordred: ^ more +2's up the whole stack please ... all will fail the diff test and need reapplying22:38
*** dboik has quit IRC22:39
*** mriedem has joined #openstack-infra22:39
clarkblifeless: have we tested 177503 with a requirements list that fails to build a wheel in the middle?22:40
*** dboik_ has quit IRC22:41
clarkbI am trying t oremember which deps were problematic so I can test22:41
clarkboh the comment says httpretty22:41
*** marun has quit IRC22:41
lifelessclarkb: httpretty delivers its own wheels now22:42
lifeless   Downloading
clarkblifeless: thats not the question though, question is, if a wheel fails to build do we build all of the remaining wheels?22:43
clarkblifeless: in the past pip did not do that22:43
*** ddieterl_ has quit IRC22:43
lifelessclarkb: pip 6.1.1 and 7.0.0 both look for errors after attempting all wheels22:43
lifelessclarkb: I do not know if we observed a bug in that loop, or if the loop did not exist in the past.22:44
clarkblifeless: the loop did not exist at one point because -r did not work22:44
clarkb(and we had confirmed that)22:44
lifelessclarkb: you misunderstand. I know the symptom.22:44
lifelessclarkb: I am saying it could have been a) the loop was buggy, and stopped early. Or b) there was no loop.22:44
lifelessclarkb: if it was a), I do not know if the loop has been fixed.22:44
clarkbsure, and I am asking have you tested latest pip?22:45
lifelessclarkb: if it was b), there is now a loop.22:45
*** jog0 has joined #openstack-infra22:45
lifelessclarkb: none of our deps fail at all now. I have not dug around for one that fails to test manually.22:45
clarkbI am tethering now, but will try to test this evening when not tethering22:46
lifelessI don't think this matters though, since right now everything works and pip 7 auto caches its own wheels on demand22:46
lifeless[and if a wheel fails to build for 7 in a way that would break the loop, it will also break pip install]22:46
clarkblifeless: it does matter if it breaks tomorrow and yesterdays code would've worked22:46
lifelessif that happens the -upstream job will also break22:47
lifelesseven if we use a loop rather than -r22:47
clarkbI don't think that makes it ok22:48
*** evgeny has joined #openstack-infra22:49
clarkbparticularly if you want ot isolate reasons for failure22:49
lifelessI do22:49
clarkb(one test failing is easier to debug than two)22:49
*** evgeny has left #openstack-infra22:49
lifelessthis is a known unknown22:49
clarkbmy known is that your change does not work22:50
lifelessthere may be any number of fails to build wheels in some special way that throws an exception22:50
clarkbuntil I have evidence of the contrary I will not approve it22:50
clarkbbut I will look for the evidence as soon as I have the ability to do so22:50
lifelessI am completely lost22:50
lifelesshere is what I know22:50
lifelessthere is some common infrastructure inside pip22:50
clarkblifeless: pip wheel -r somefile will bail out early if a wheel fails to build22:50
lifelessclarkb: no, it won't22:51
*** whoops has quit IRC22:51
clarkbthis was a known as of the lsat time I did anything with wheels and pip (which is admittedly some time)22:51
lifelessthat infrastructure is used in pip wheel22:51
clarkblifeless: ok please show me evidence of that22:51
lifelessand its used in pip install in 7.0 to autocache wheels22:51
clarkblifeless: that is all I am asking for and will go find it myself as soon as I am able22:51
lifelessthere is a test in 7.0 for pip install with a thing that will fail to build a wheel22:51
*** annegent_ has quit IRC22:51
lifelessso I know that with a vanila fails-to-build-a-wheel that the loop won't be interrupted22:52
lifelessand all the wheels will be attempted22:52
clarkblatest pip is still 6.1.122:52
clarkbnot 7.022:52
lifeless6.1.1 has the same code22:52
lifelessand it has not changed22:52
*** jog0 is now known as jogo22:52
clarkbok so now all I need is to test it22:52
lifelessit just doesn't have the second user - the autocached22:52
clarkbbecause 6.something definitely did fail22:52
mtreinishmordred: if you wanted big numbers about the gate for slides:
lifelessthere may be ways that a wheel can fail to build that will provoke a failure in the loop22:52
lifelessI don't know of any22:53
mtreinishsince we're basically at the end of the cycle that's basically a cycle's worth of tempest tests22:53
lifelessbut its a big universe :)22:53
clarkblifeless: the previous case was httpretty22:53
clarkblifeless: which works now because they ship a wheel22:53
lifelessthe httpretty case will work22:53
clarkblifeless: but not becaues of pip22:53
lifelessbecause of pip22:53
clarkbno because they ship a hweel22:53
clarkbso pip wheel is a noop22:54
lifelessif you use a requirements file that clamps down to an earlier httppretty22:54
lifelessit will work22:54
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: Add dstat to subnode in multinode mode
*** shashankhegde has quit IRC22:54
*** mwagner_lap has joined #openstack-infra22:54
lifelesssometing that sends SIGINT to its parent process wouldn't22:56
*** jamesmcarthur has quit IRC22:56
*** sdake_ has quit IRC22:57
clarkbok confirmed that latest pip 6.1.1 with httpretty==0.8.0 will fail to build an httpretty wheel but then continue to build a wheel for tox22:57
*** jtriley has joined #openstack-infra22:58
*** signed8bit is now known as signed8bit_ZZZzz23:01
SpamapSSo, google is no help in squelching these, anybody:23:01
*** Sukhdev has quit IRC23:01
SpamapS/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see
SpamapStime to upgrade to utopic/vivid ?23:02
clarkbSpamapS: 1) cry 2) you can get a new platform with latest python2.7 3) install requests[security] #note this requires a bunch of C headers for pyopenssl23:02
greghaynesSpamapS: There is a logging.conf filter23:02
greghaynesI know ive seen a filter on a patch somewhere23:02
greghaynesI think it was in gertty23:02
clarkbya I think it was gertty23:02
clarkbbecause it broke curses23:03
lifelesswhich will tell you that you need pyopenssl ndg-httpsclient  pyasn123:03
SpamapSfound this one too23:04
lifelessaka this is why 'releases' are bad23:04
clarkbSpamapS: also note that about 80% of requests installs are affected based on rough graph provided by dstufft23:04
SpamapSseems like should be updated to explain the libs to silence that23:05
SpamapSclarkb: osssuummm23:05
clarkbalso requests bundles urllib3 so this has to be fixed in two places23:05
clarkb* the spamming must be fixed in two places I think its reasonable to provide one warning per process invocation23:06
clarkbbut every request is a bit over the top23:06
*** shashankhegde has joined #openstack-infra23:06
dstufftclarkb: I think 2.6.1 only does it once23:06
SpamapSyeah keystoneclient had the same insanity23:06
SpamapSlogs filling for days becuase of the premature v2 deprecation23:07
*** yamamoto has joined #openstack-infra23:07
*** armax has joined #openstack-infra23:07
*** hemna is now known as hemnafk23:11
lifelessclarkb: many places - see the pip bug I linked23:11
fungiclarkb: nibalizer: was there a question at some point about whether puppet assumes file ownership from the puppetmaster's filesystem when not specified in the file resource itself? if so, i have specific evidence i can share that it definitely does this23:12
clarkbfungi: not from me at least (its totally viewable when you use different puppet envs because perms change)23:12
fungiyep, okay... just checking23:13
lifelessmordred: / fungi: can you review this stack - ?23:13
jogolifeless: reading enjoying it so far23:14
nibalizerfungi: yep23:15
lifelessjogo: cool23:15
nibalizerthis is why File [ owner => root, group => root,] ends up at the top of site.pp a lot23:15
clarkbjogo: that test fail you linked is crazy fwiw, ssh works then stops working23:15
clarkbjogo: guesses are cloud networking issue, sshd dies, or keys are deleted23:16
jogoclarkb: yeah, I am wondering if that is a really strange one off or not23:16
jogoonly 1 hit in 61 or so jobs23:16
jogoclarkb: mainly shared it so you are aware23:16
mtreinishclarkb: or someone sneezed in the data center and the network cable fell out23:16
jogolifeless: sounds like you have some serious pipping to do23:16
fungiproviders need to start investing in ieee sneezeproof cabling23:17
clarkbpossible race in network setup maybe? iptables/ovs23:18
lifelesshow do you get unprocessed logs?23:18
clarkbbut I think it happens before that23:18
clarkblifeless: accept txt/plain and there is a parameter too23:18
clarkblifeless: wget willget nuprocessed fwiw. let me dig up param23:19
lifelessah cool23:19
lifelessI am wgetting23:19
lifelessits just 116M23:19
lifeless{something assploded}23:19
lifelessclarkb: don't worry, wget DTRT is enough for me23:19
lifelessjogo: there's two conceptual lines of work23:19
lifelessjogo: three bugs, and it all folds into one specific interaction23:20
*** sabeen1 has quit IRC23:20
*** dimtruck is now known as zz_dimtruck23:20
*** wenlock_ has quit IRC23:20
*** yamamoto has quit IRC23:21
jogolifeless: right23:21
jogolifeless: to be clear, the goal is:  a single constraints file for all of openstack23:22
jogothat uses the '==' notation23:22
jogowhat do you mean loosen up the constraints in all our release branches23:22
lifelessjogo: so we have olso.messaging >= 1.4 < 1.523:23
lifelessjogo: it needs to be at minimum >= 1.4 < 1.623:23
lifelessjogo: otherwise we can't upgrade without violating the constraint23:23
lifelessand once pip 2687 is fixed, that will error23:24
jogook, how does that tie into requirements.txt vs setup.cfgs install_requires or whatever that was?23:24
lifelessso requirements.txt in each repo goes away23:25
jogolifeless: also so your saying teach pip to support a requirements file and a seperate constraints file?23:25
lifelessthe install_requires it contained move into setup.cfg23:25
lifelessthis can be done independently though, its really just aligning with upstreams expectations23:25
lifelesswe could perhaps have a constraints in each repo with ==, but I really question the point of doing that since we would have to be using the central one in all CI anyhow23:26
fungilifeless: out of curiosity, once we have olso.messaging>=1.4,<1.6 how do we handle olso.messaging jumping from 1.4.7 to 2.0.0?23:26
lifelessfungi: so thats a true API break happening23:26
lifelessfungi: from a deployment perspective, that means you can't do an incremental deployment across that boundary23:27
*** kaisers has quit IRC23:27
lifelessfungi: unless you can upgrade all the users of it first to accept it23:27
fungigot it, so assume a requirements list update to accommodate those situations23:27
lifelessand point releases of the stable servers23:27
*** emagana has quit IRC23:28
lifelesse.g. liberty .4 release with < 2.1 in its deps23:28
lifelessyou deploy those, that loosens the constraints, you deploy the thing that will bring in 2.0.023:28
jogowe also need to make some guarantees about compatibility with our prior release in our library development23:29
lifelessYou can only do that once you know that liberty is compatible with 2.0 of course23:29
lifelessyup. Sorry devs, inconvenient truth. :)23:29
jogolifeless: that line could be expanded on I think. I am not convinced that will work 100% of the time23:29
*** otter768 has joined #openstack-infra23:30
lifelessjogo: go on23:30
jogolifeless: well what fungi said is part23:30
jogolifeless: but what about non-openstack owned libraries23:30
jogoI also question the requirement actually23:30
jogobeing able to partially upgrade a single box?23:30
lifelesswe do this in grenade23:31
fungisort of23:31
openstackgerritMerged openstack-infra/system-config: Log #openstack-neutron-ovn.
clarkbbut its a turn it all off then upgrade upgrade23:31
fungiwe upgrade the services while keeping the requirements as they were23:31
fungiand the configuration as it was23:32
lifelessfungi: since the requirements are partitions, I don't understand that statement23:32
jogofungi: we actaully don't keep the old requirements anymore AFAIK23:33
fungioh, poop23:33
*** annegentle has joined #openstack-infra23:33
fungii missed when that changed23:33
* jogo double checks23:33
*** wenlock has quit IRC23:33
fungiused to be install a stable/last devstack, then shut it down, then reinstall current git trees with stable/last requirements list23:34
lifelessthats the ultimate form of backwards comapt23:34
fungithough i also may have simply dreamt that23:34
lifelessI think its harder than needed23:34
clarkbfungi: there are a few jobs in the check queue that have been running for hours23:34
lifelessjogo: what about the partial things where we bring up new nova conductor, old compute23:34
jogolifeless: we don't restart the old service23:34
*** otter768 has quit IRC23:35
fungiclarkb: different jenkins masters too23:35
jogoso it uses the old copies23:35
jogoits a hack23:35
clarkbfungi: ya looks like maybe related to the swift uploads?23:35
lifelessjogo: and hope that when its starts new processes via rootwrap, that it all works23:35
jogolifeless: yup23:35
lifelessjogo: aiee23:35
clarkbERROR:root:File posting error23:35
clarkbwhich I think is the get console log from jenkins post?23:35
*** annegentle has quit IRC23:35
fungiclarkb: yeah, but for example last tried at 19:33 utc and has just been sitting there ever since23:36
*** kaisers has joined #openstack-infra23:36
clarkboh wait no thats swift23:36
clarkbI wonder if that is what happens when we exceed the time limit on the tempurl?23:36
clarkbjhesketh: ^ you may be interested in this23:36
fungilooks like we _could_ abort these if we want, but being able to catch them in action seems like a useful reason to not do that23:37
*** yamada-h has joined #openstack-infra23:37
clarkbthough it specifically says network unreachable23:37
clarkbhopefully we would get not authorized if tempurl is no longer valid23:38
clarkbnotmyname: mattoliverau ^23:38
jheskethclarkb: hmm, I'll take a look in a bit23:38
jogolifeless fungi:
notmynameclarkb: tempurl expires time should be at the time of the request, not the time the request completes23:39
clarkbnotmyname: right but would we get "network unreachable" as the error if it had expired?23:40
notmynamethat doesn't sound right to me23:40
fungihrm, nodepool's getting EOFError testing ssh to devstack-precise nodes in hpcloud. not having a problem with bare-precise there, nor is rax having issues with devstack-precise23:40
notmynameif it had expired, you'd see a 401 response23:40
clarkbnotmyname: thanks23:40
fungisomething must be not quite right with those images23:41
*** erlon has quit IRC23:41
lifelessjogo: right so that is doing precisely what I said23:41
lifelessjogo: its loosening the constraints23:41
lifelessjogo: but we can't do that for the things we're consuming as releases23:41
lifelessjogo: so we have to make releases of them with looser constraints23:42
*** yamada-h has quit IRC23:42
jogolifeless: its syncing constraints with g-r23:42
jogolifeless: from master23:42
jogolifeless: some of that is loosening some isn't that is besides the point here23:42
jogohmm although it is using pip install -e23:44
jogowill that upgrade deps that conflict?23:45
jogolifeless: kk thought so23:45
lifelessand with 2687 fixed, fail if it can't resolve23:45
jogolifeless: so going back, I don't think supporting stable/N services + stable/N+1 services on the same node is a reasonable requirement23:46
*** dims has quit IRC23:46
clarkbjogo: thats how you upgrade a service at a time23:46
jogoits a nice one to have but we don't aim for that today23:46
clarkbeg upgrade keystone first23:47
clarkband work your way up the stack23:47
clarkb(or down, I don't actually know what the appropriate direction would be)23:47
jogoclarkb: we upgrade one at a time by knowing if you don't restart the service it uses the old version already in memory23:47
clarkbjogo: what happens if you have to reboot23:47
clarkbjogo: thats not an acceptable method23:47
jogoclarkb: your fucked23:47
jogoclarkb: my point is we don't suppor that now23:48
lifelessjogo: ok so in summary: this thing I propose to avoid users getting fucked we currently say they can get fucked on?23:48
jogoI don't think we should add that as a hard req for fixing pip23:48
lifelessjogo: can we assess it on its merits rather than what we do today ? :)23:48
jogolifeless: merits and cost yes23:48
*** jtriley has quit IRC23:48
jogolifeless: if we can support this with minimal overhead and risk by all means lets do it23:49
jogojust don't see that being small23:49
lifelessso previously we didn't cap the libraries AIUI23:49
jogoanyway I believe sdague and dtroyer know more about this then I do23:49
jogoso I defer to them for further insight23:49
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Update to search UI.
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Renamed result-set-size directive
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Result set paging update.
lifelessmy memory is that we started capping them because we made mistakes23:49
lifelessand broke ourselves23:49
lifelessnot that we intended to do so23:49
jogolifeless: correct, other things also broke us as well23:49
jogobesides ourselves23:49
*** armax has quit IRC23:49
jogo btw for your blog post an example of the final result would be neat23:50
jogosomething like the different reqs files and how they get used23:50
lifelessso the design here is to avoid that harming CI by having complete centralised control23:50
jogolifeless: ++23:50
*** sarob has quit IRC23:51
lifelessand if we have something that would break stable get out into the wild, we can find and fix without everything wedging23:51
jogoso I think the possible downside of requiring backwards compat with old servers is:23:51
clarkblifeless: so master would remain open?23:51
clarkbif so that wasn't clear in the blog23:51
openstackgerritKhai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server
lifelessclarkb: what do you mean by open ?23:52
jogorequiring extra work and planning to prevent backwards incompatible  breaks23:52
clarkblifeless: I mean master wouldn't pin everything and instead plumb the depths looking for broken23:52
jogofor openstack owned and 3rd party libraries23:52
clarkbthen at rc time we can pin23:52
jogoclarkb: you see this ?23:52
clarkbjogo: yes that is what I am referring to23:52
lifelessclarkb: I think that is something we could play around with23:52
lifelessclarkb: we could constrain master precisely to known good as the blog post shows23:53
*** sarob has joined #openstack-infra23:54
*** yamamoto has joined #openstack-infra23:56
lifelessclarkb: or we could unconstraint master and ride the wave23:56
lifelessclarkb: I think having master constrained and reporting on potential upgrades is better23:56
*** armax has joined #openstack-infra23:56
lifelessclarkb: because we spend a lot of time in firedrills23:56
clarkbexcept they are often real bugs23:56
clarkbso by ignoring them on master we just shift the work23:57
lifelessI didn't say ignore23:57
clarkbso I guess it is a question of whether or not shifting that work is beneficial23:57
clarkblifeless: I know23:57
clarkbbut history has shown it will be ignored. I will be pleasantly surprised if we do otherwise23:57
lifelessclarkb: interrupting 1000 developers is very expensive23:57
clarkblifeless: but its rarely a real interrupt23:57
lifelessclarkb: except when it is23:57
jogoclarkb lifeless: OpenStack has grown up a lot since 'history'23:57
clarkbcode got -1 because openstack is broken23:57
*** Somay has joined #openstack-infra23:57
clarkbthats a real interrupt23:58
jogonot sure if we can use said data as a useful reference anymore23:58
lifelessclarkb: code can't land is also an interrupt23:58
clarkbjogo: I don't think it has really learned its lessons23:58
jogoeither way, this is something we can easily adjust over time23:58
clarkbjogo: we still can't turn on ipv6 in devstack for example23:58
lifelessindeed, its flexible23:58
jogoclarkb: that is a very different type of issue23:58
clarkbjogo: not really, we are talking about not putting off bugs to make now work23:58
jogobut yeah we are really arguing over what would amount to a few lines of configuration that is easy to cahnge over time23:58
clarkbjogo: its the same type of bug23:59
clarkbif we just turned on ipv6 and interrupted everyone then this would've been fixed in kilo23:59
clarkbinstead nova ignored it23:59
jogoclarkb: most of the dependency bugs are fairly isolated in nature, is IPv6 thsame23:59
clarkband now its another cycle until we can tet this23:59
jogoalso part of the issue is rallying a team around a problem23:59

Generated by 2.14.0 by Marius Gedminas - find it at!