Wednesday, 2013-12-04

jeblairfungi: excellent, i'll leave 58106 for you to aprv when you feel like this conversation has successfully concluded :)00:00
*** dstanek has quit IRC00:00
fungijeblair: my pleasure00:00
fungizaro: your pending python-jenkins contribution shouldn't hold up approval of right?00:01
mordredlifeless: that's right00:01
*** loq_mac has quit IRC00:01
lifelessmordred: that seems undesirable00:01
mordredlifeless: but I hope to soon get some build advice to the pip guys to let them know if a release will break us00:01
lifelessmordred: perhaps telling it to install from the mirror would avoid this?00:01
mordredlifeless: the version of pip you should always use is the latest version - all old versions of pip are broken00:01
mordredlifeless: perhaps00:02
lifelessthough mirror as version lock is an antipattern00:02
mordredwe could tell git pip to install pip==1.4.1 if we wanted00:02
mordredand point that at the mirror00:02
lifelessanyhow, thanks for explaining. This isn't at all obvious, and I think you should capture that somewhere in the patch.00:02
lifelessfor our future sanity.00:02
mordredwell, the real guts are in devstack, not here00:02
mordredthis is just enabling the flag00:03
mordredI agree though - I believe that more sanity should be captured00:03
*** rongze has quit IRC00:04
fungicompletely unrelated, btw, but i deleted several vacated cinder volumes from static.o.o a little while ago, replaced those with larger ones, and so far nothing seems to have broken horribly so the plan there is going well (continuing to shuffle extents to the larger volumes but i expect it to be going on for days)00:05
jeblairfungi: thank you00:05
fungilinux lvm2 is a marvellous work of engineering00:06
*** dcramer__ has quit IRC00:06
lifelesswe layer lvm on top of cinder vols?00:06
lifelessdo we do redundancy that way too?00:07
funginope, just jbod00:07
lifelessso - cinder then mdadm then lvm ?00:07
*** datsun180b has quit IRC00:07
lifelessor we trust that cinder will be available ?00:07
funginope, no md-raid, just lvm2 on top of cinder volumes00:07
lifelessok, and just linear in lvm2?00:08
openstackgerritA change was merged to openstack-infra/config: Add gate-solum-devstack job
openstackgerritA change was merged to openstack-infra/config: Add tooz
*** flaper87 is now known as flaper87|afk00:09
fungilifeless: rackspace limits the size of a cinder volume to 1tb, their underlying tech (xen) limits block devices per vm to 16 (two of which are already preclaimed by their model), and they limit (presumably artifically) aggregate cinder storage to 10tb per account00:09
jeblairlifeless: it seems likely that raid is happening behind cinder; but we aren't sure, so attempting to raid on top of cinder (without knowing what's under it) could mean we detune the storage instead of tuning it.00:09
*** reaper has quit IRC00:10
*** ^d has joined #openstack-infra00:10
jeblairfungi: oh, are we going to hit the 10tb limit?  i'm assuming we could get that raised.00:10
fungiright, at $oldjob-2 we always had to explain to our cloud customers that doing raid-1 on top of our san raid-1 wasn't especially sane00:10
jeblairfungi: exactly :)00:11
fungijeblair: i only need ~9tb to reach the discussed goal, but yeah their faq makes it sound like a simple quota00:11
*** wenlock has quit IRC00:11
jgriffithfungi: not to jump in but mdraid between two cinder volumes is something I believe RAX recommends for certain cases00:12
fungijgriffith: oh, cool. however in our case i think we just need $lots_of_hopefully_reliable_storage until we develop a better object storage solution to swap it into00:13
*** rcleere has quit IRC00:13
jgriffithfungi: ahhh got ya00:14
jgriffithfungi: stick with block :)00:14
*** boris-42 has quit IRC00:14
fungiso aggregating with lvm2 is an easy solution for now00:14
jgriffithfungi: totally makes sense00:14
fungijgriffith: i don't think we'll ever be getting away from block storage enturely, just some bulk storage for applications where it makes sense to use object stores00:15
jgriffithFWIW I have a bp somewhere to create that as an optional volume type in Cinder00:15
fungioh, that would be nifty00:15
jgriffithfungi: Yeah... and if I could get descent perf out of clustered LVM or LINBIT I'd like to make progress on that as well00:15
jgriffithbut anywho00:15
*** melwitt has quit IRC00:16
*** nati_ueno has joined #openstack-infra00:18
*** nati_uen_ has quit IRC00:21
jeblairclarkb: yes, i think it's that simple. (requirements update)00:21
*** slong has quit IRC00:22
*** slong_ has joined #openstack-infra00:22
openstackgerritA change was merged to openstack-infra/gear: geard: wrong reference to setup_logging()
*** vipul is now known as vipul-away00:27
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues.
openstackgerritA change was merged to openstack-infra/config: Send clouddocs-maven-plugin events to doc channel
*** matsuhashi has joined #openstack-infra00:31
openstackgerritA change was merged to openstack-infra/config: Remove remote-directory from publishing in api-jobs
*** sarob has quit IRC00:32
*** wabat has left #openstack-infra00:33
*** nati_ueno has quit IRC00:33
*** sarob has joined #openstack-infra00:33
*** ^d has quit IRC00:33
openstackgerritA change was merged to openstack-infra/config: Move ttx/os-releasing to os-infra/release-tools
openstackgerritA change was merged to openstack-infra/config: Enable #openstack-climate IRC channel logging
*** sarob has quit IRC00:37
*** dstanek has joined #openstack-infra00:38
openstackgerritA change was merged to openstack-infra/config: Add oslo.messaging to release status tracking
*** vipul-away is now known as vipul00:40
*** balar has joined #openstack-infra00:41
clarkbI am no longer afk00:43
clarkbEmilienM: for grenade no, we haven;t sorted that out yet00:43
clarkbI think grenade is still in a state of flux, we should sort it out, but I am not going to -1 changes to grenade jobs that don't follow the devstack job templating00:44
clarkblooking at the zuul fix now00:45
clarkbfungi: yes thank you (re commit message update)00:46
*** openstack has joined #openstack-infra00:46 [freenode-info] channel flooding and no channel staff around to help? Please check with freenode support:
openstackgerritA change was merged to openstack-infra/gear: Correctly set job known/running attributes
openstackgerritA change was merged to openstack-infra/git-review: Ignore content of merge commits in reporting
*** salv-orlando has quit IRC00:53
clarkbjeblair: I am trying to understand the zuul change. I grok the intent, but I don't see where the shared state is carried across each iteration of the new loop00:55
clarkbjeblair: does the argument to combineChangeQueues need to be last_change_queues instead of change_queues?00:55
*** sarob has joined #openstack-infra00:55
*** openstackgerrit has quit IRC00:56
*** openstackgerrit has joined #openstack-infra00:56
clarkbjeblair: I think maybe the code works there because one extra pass is made?00:57
clarkbI will write notes shortly00:57
*** mrodden1 has joined #openstack-infra01:02
*** mrodden has quit IRC01:03
*** dims has quit IRC01:04
jeblairclarkb: i think you're on to something.  :)01:10
clarkbjeblair: notes just about done01:10
clarkbjeblair: and posted01:11
*** sarob has quit IRC01:11
*** sarob has joined #openstack-infra01:12
*** reed has quit IRC01:12
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues.
jeblairclarkb: thanks, nice catch01:13
clarkbjeblair: I think there is still a small problem in that patchset :)01:14
jeblairclarkb: hehe01:15
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues.
clarkband posted a review01:15
clarkbthere we go01:15
clarkbI am +2 as soon as jenkins is happy with it01:16
*** sarob has quit IRC01:16
*** mfer has joined #openstack-infra01:18
clarkbfungi: any reason for not approving ? I am going to approve it shortly if you don't have strong feelings about it01:18
*** mfer has quit IRC01:18
*** dims has joined #openstack-infra01:19
fungiclarkb: since it was something mordred had originally requested i figured he might want to have a look too, but i'm good with merging it01:19
*** krotscheck has quit IRC01:19
clarkbfungi: I really like the info it gives us, and we can tweak later if it needs to do more or different things01:20
fungialso, i mostly don't aprv patches to pbr which mordred has not at least +2'd, but this is tests only01:20
jerryzclarkb: i notice that latest elastic- recheck's comment will include recheck/reverify bug #, won't this trigger a rebuild automatically?01:21
clarkbya if this was pbr proper I probably would defer to him but its in the test and informational only so +A01:21
clarkbjerryz: no, the strings to recheck/reverify are very specific and must be 'recheck bug XXXXXX' basically01:21
fungijerryz: zuul only considers it if it's the *only* text in the comment01:21
fungithe elastic-recheck comments are taking that into consideration by design, because they want an actual human to make sure that's sane before adding it as a separate comment01:22
*** julim has quit IRC01:22
openstackgerritClark Boylan proposed a change to openstack-infra/zuul: Ignore .venv directory
clarkbI am going to self approve ^ it is a simple tiny change that needed to be detached from its parent so that gate testing could happen01:24
clarkbtrivial rebase detection picked up on that \o/01:24
fungii can review01:24
fungilgtm, approved01:25
jerryzfungi clarkb: i see. thanks01:25
clarkbfungi: danke01:25
funginothing like approving a one-line change to make me feel productive, after all ;)01:26
*** nati_ueno has joined #openstack-infra01:26
openstackgerritClark Boylan proposed a change to openstack-infra/config: Enable requirement update proposal jobs
openstackgerritA change was merged to openstack-infra/zuul: Ignore .venv directory
*** nati_ueno has quit IRC01:32
*** nati_ueno has joined #openstack-infra01:32
*** nati_ueno has quit IRC01:33
*** kui has joined #openstack-infra01:33
*** nati_ueno has joined #openstack-infra01:33
*** nosnos has joined #openstack-infra01:35
*** jamesmcarthur has joined #openstack-infra01:35
*** nosnos_ has joined #openstack-infra01:37
*** nosnos has quit IRC01:40
*** wenlock has joined #openstack-infra01:47
openstackgerritClark Boylan proposed a change to openstack-infra/config: Run the mirror if we land changes to pypi-mirror
clarkbanother change that needed rebasing to be mergable01:48
clarkbI am trying to get changes that have been approved through01:48
fungiapproving that one once jenkins reports back01:49
clarkbwoot thanks (I don't think it was a trivial rebase according to gerrit but really it is)01:49
fungiyeah. it's a 3-line change no matter how you slice it01:49
fungisorry, 4 line01:49
*** mfer has joined #openstack-infra01:50
clarkbjog0: is that problem familiar? I am not finding a bug for it (but haven't looked beyond console log yet)01:51
*** Ryan_Lane has joined #openstack-infra01:51
openstackgerritA change was merged to openstack-infra/config: Run the mirror if we land changes to pypi-mirror
*** dstanek_afk has joined #openstack-infra01:55
*** Ryan_Lane has quit IRC01:56
fungii've similarly been trying to set first priority on changes which already have at least one +2 so we can get them cleared out02:00
*** rongze has joined #openstack-infra02:01
*** rongze has quit IRC02:01
clarkblooks like maybe tempest is failing to see a server as active? I am going to do that sad thing and submit a bug against tempest02:01
*** rongze has joined #openstack-infra02:02
*** bingbu has joined #openstack-infra02:04
clarkbI have reverified 48803 with a bug. Going to walk home now02:07
fungiahh, bug 125756102:07
uvirtbotLaunchpad bug 1257561 in tempest "test_create_backup test not noticing that server has become active" [Undecided,New]
fungiclarkb: is not going to merge. i'll rebase it02:09
fungier, i meant 4880302:09
fungi(stupid paste buffers)02:10
clarkbgah, I should've checked that... oh well and thanks02:10
clarkbnow I am really walking home02:10
fungiyeah, it's hanging out at the back of the gate until i get this rebase submitted, then i'll approve once it passes check02:11
fungispecifically, it's conflicting with 54639 (currently ahead of it in the gate_02:14
zarofungi: approve away on 5641202:14
fungizaro: thanks--done02:15
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Deprecate projectName setting of CopyArtifact
*** ruhe has joined #openstack-infra02:18
*** elasticio has joined #openstack-infra02:18
openstackgerritJeremy Stanley proposed a change to openstack-dev/pbr: Use wheels for installation
*** dims has quit IRC02:23
*** Abhishek has joined #openstack-infra02:24
*** fungi has quit IRC02:24
Abhishekhey guys, I am trying to write a new gating job02:24
Abhishekhow do I install a python module for the job?02:24
Abhishekis it okay to just do a pip install in a bash script?02:25
*** fungi has joined #openstack-infra02:26
*** guohliu has joined #openstack-infra02:27
mesteryHowdy folks, anyone else noticing the openstack-dev and openstack lists appear to be borked?02:27
mesteryI haven't received emails on either since approx. 2:35PM CST or so.02:27
mesteryFor reference, this is the last email I received from openstack-dev today:
fungimestery: i've received messages from them as recently as 2013-12-04 02:04:41 utc02:31
mesteryfungi: If you look at the archives, since the email I just referenced there have only been about 20 more.02:31
mesteryThat's curiously low for openstack-dev to me.02:31
mesteryfungi: I checked with one other Cisco person who also hasn't received any since the same time as me.02:32
mesterySo unsure if it's a Cisco problem or not ...02:32
mesteryfungi: Likely a Cisco problem of some sort perhaps.02:32
fungii've been receiving quite a few on both lists you mentioned since that time02:32
mesteryfungi: thanks for the point of reference!02:32
* mestery keeps digging and makes a note to move his list traffic off of cisco's servers going forward.02:32
fungiyeah, the most recent messages in the -dev archive look familiar (though i've already deleted my copies, i could match message ids from my mta logs)02:33
* fungi checks the delivery queue on the listserv for cisco domains02:34
mesteryfungi: Thanks for the help!02:34
*** rwsu has quit IRC02:35
*** esker has joined #openstack-infra02:35
fungimestery: aha!02:39
fungi2013-12-04 02:23:03 1Vo27D-0003zX-IO ** R=dnslookup T=remote_smtp: SMTP error from remote mail server after RCPT TO:<>: host [2001:420:1201:6::ad25:93e6]: 550 Connections from the host (2001:4800:780e:510:3bc3:d7f6:ff04:b736), originating from SenderBase Network Owner ID: None, are being rejected due to a low SenderBase02:39
fungiReputation Score. See for more information or contact your IT support team.02:39
*** rwsu has joined #openstack-infra02:39
mesteryfungi: So it looks like we're blocking it, thanks for the tip.02:39
fungimestery: senderbase was bought by ironport was bought by cisco, right?02:39
fungior am i misremembering?02:40
mesteryfungi: No, you've remembered correctly.02:40
mesteryfungi: I'll file an IT case internally. *sigh*02:40
* fungi goes back to not being a network ops guy02:40
* mestery grumbles about just wanted to get work done.02:40
mesteryfungi: Heh, thanks for the help!02:40
jamesmcarthursdague: hi - we're currently trying to troubleshoot the issues you're having with the profile page.02:41
fungiif your internal ticket happens to mention that senderbase should give openstack servers a better reputation, especially the ones hosting technical mailing lists which have never sent me any spam, that would also rock02:41
fungimestery: ^02:41
mesteryfungi: Got it, will do.02:41
jamesmcarthursdague: was wondering if you could provide any other info besides what you sent to Jonathan and Todd.02:42
fungijamesmcarthur: excellent--we've gotten several reports, of broken in the affiliations/contact profile bits, so it's appreciated02:42
jamesmcarthursdague: having trouble replicating the problem on all browsers without actually cutting off our internet connection02:42
jamesmcarthurfungi: cool. well i'm happy to field questions/take info directly.02:42
jamesmcarthurfungi: we'd like to get this nailed down ASAP02:43
fungijamesmcarthur: i'll try to remember to send them your way. there were a couple who said they were unable to change their contact addresses without first adding an affiliation in their profile. that's teh main one which comes to mind02:43
jamesmcarthurfungi: we did find another error with the DOM that we're updating tomorrow, but I'm not 100% that's related to our issue here.02:44
fungijamesmcarthur: great. i'll keep an ear to the ground02:44
jamesmcarthurfungi: the best thing to do is submit a ticket to
fungiand will send them there. i didn't know about that lp project existing, so thanks for the tip!02:44
jamesmcarthurfungi: we've got our eyes on this now and will be monitoring/reviewing/addressing bugs that come in for and will be trying to stay on top of things from here forward02:45
jamesmcarthurfungi: also, do you sleep? you're online all the time ;)02:45
fungijamesmcarthur: i'm working on jacking an internet into my skull so i can irc WHILE i sleep. putting off further sleep until that's done02:46
*** marun has joined #openstack-infra02:46
mgagnejamesmcarthur: he installed an irc client for google glass02:46
jamesmcarthurmgagne: is that true? because that would be quite cool.02:46
fungimgagne: close. i have my tmux session via mosh from a beagleboard plugged into a pair of video goggles02:46
fungi(no joke)02:47
mgagnesee? :D02:47
jamesmcarthuri feel like i'm not trying hard enough now ;)02:47
* mgagne got lucky02:47
jamesmcarthurbut seriously, that's awesome02:48
fungiyou couldn't try less hard than me. i'm currently watching star trek on blu-ray02:48
fungiit's slack time for me right now02:48
*** MarkAtwood has quit IRC02:49
*** Ryan_Lane has joined #openstack-infra02:51
*** Abhishek has quit IRC02:52
*** Ryan_Lane has quit IRC02:55
*** jaypipes has quit IRC02:57
openstackgerritMathieu Gagné proposed a change to openstack-infra/config: Remove puppet-openstack_dev_env from StackForge
*** ruhe has quit IRC03:01
*** vkozhukalov has joined #openstack-infra03:09
*** elasticio has quit IRC03:12
*** MarkAtwood has joined #openstack-infra03:12
*** MarkAtwood has quit IRC03:13
*** MarkAtwood has joined #openstack-infra03:13
*** vipul has quit IRC03:15
*** vipul has joined #openstack-infra03:16
*** soren has quit IRC03:17
*** rainya has quit IRC03:17
*** adam_g has quit IRC03:17
*** antonym has quit IRC03:18
*** harlowja has quit IRC03:19
*** rainya has joined #openstack-infra03:19
*** Shrews has quit IRC03:19
*** pvo has quit IRC03:19
*** Shrews has joined #openstack-infra03:19
*** antonym has joined #openstack-infra03:20
*** pvo has joined #openstack-infra03:22
*** soren has joined #openstack-infra03:23
clarkbI think we need to make hpcloud region a and b devstack-precise-check only03:27
fungislow still?03:28
*** mfer has quit IRC03:29
clarkbfungi: ya03:30
clarkbfungi: is holding up the gate03:30
clarkbslightly afraid it is going to hit a timeout03:31
clarkbmordred: ^ how do we give that feedback to HP?03:31
*** esker has quit IRC03:34
*** rcleere has joined #openstack-infra03:34
clarkbgah it timed out03:34
clarkbit almost made it through03:35
*** gyee has quit IRC03:35
clarkbgoing to propose the change to check pipeline only them03:35
clarkbthat is frustrating03:35
fungiwell, rs iad didn't prove out faster either03:37
openstackgerritClark Boylan proposed a change to openstack-infra/config: Don't use new HPCloud regions in the gate.
clarkbfungi: right, but this is a step backwards for hpcloud03:38
fungior a step forward, if one of the goals was to prevent tenants from consuming too many resources03:39
clarkbstepbackward for us as customers :)03:40
fungiso true03:45
*** loq_mac has joined #openstack-infra03:45
zarofungi, clarkb : i'm thinking about doing gerrit upgrade tomorrow. i think i need access to review-dev right?03:46
*** Ryan_Lane has joined #openstack-infra03:51
fungiwe have review-dev under bup, right? so if an in-place upgrade via puppet went horribly wrong, we could presumably restore it in the worst possible case03:53
clarkbyes I think so03:54
clarkbzaro: no, it can all be done via puppet03:54
fungizaro: what is the upgrade path from 2.4 to 2.8?03:54
zarogreat!  i'm pretty sure you can go 2.4 -> 2.803:54
zaroi'll double check though.03:55
*** amotoki has joined #openstack-infra03:55
zarowhat's the restore mechanism?03:55
*** Ryan_Lane has quit IRC03:56
fungithe restore mechanism is one of the root users invoking bup restore, then moving files into place and reloading the db from the last mysqldump in the backup03:57
fungi(in other words, manual)03:58
zarowhat's the script in puppet that can do a gerrit upgrade?03:59
clarkbzaro: there isn't a script, you just change the war file03:59
clarkbit is all done through puppet04:00
zaroneed to do more than just change out the file.  i think you need to run a command to do the upgrade.04:01
clarkbpuppet takes care of it04:01
clarkbyou have to run the command that does DB migrations04:02
*** herndon_ has quit IRC04:06
*** nati_ueno has quit IRC04:08
zaroi see the gerrit wars are hosted on tarballs.o.o04:09
clarkbzaro: right so that is why jeblair was talking about creating a 2.8 branch and putting your patch on it so that we would get a war built for it04:09
zaroclarkb, fungi : do i have access to put new files on tarballs?04:09
clarkbzaro: no, they are put there by jenkins when changes are proposed to and merge on our gerrit repo04:10
* zaro gets it now04:11
zaroohh wait.  2.8 branch would require new tooling to build.04:11
clarkbbecause of buck?04:12
zaronewer gerrit uses buck instead of maven04:12
clarkbso we may need to tackle that first04:12
zarowould that mean new job to build as well?04:12
zaroso new job that builds gerrit 2.8 branch?04:14
zaroi believe buck is build from source as well.  i don't think there are packages for buck yet.04:14
*** CaptTofu has quit IRC04:15
*** CaptTofu has joined #openstack-infra04:16
*** basic` has quit IRC04:17
*** basic` has joined #openstack-infra04:18
*** basic` has joined #openstack-infra04:18
*** MarkAtwood has quit IRC04:23
*** jerryz has quit IRC04:23
clarkbzaro: it might require different jobs, zuul can figure out which one to run based on the branch04:28
*** prad has quit IRC04:33
*** chandankumar has joined #openstack-infra04:36
*** loq_mac has quit IRC04:37
*** nosnos has joined #openstack-infra04:38
*** nosnos_ has quit IRC04:42
mordredmaybe we should try just using the upstream war?04:45
*** rongze has quit IRC04:45
*** StevenK has joined #openstack-infra04:45
*** Ryan_Lane has joined #openstack-infra04:51
*** FallenPegasus has joined #openstack-infra04:52
clarkbmordred: we could try that with a build off of master04:52
*** Ryan_Lane has quit IRC04:56
*** SergeyLukjanov has joined #openstack-infra04:56
*** SergeyLukjanov is now known as _SergeyLukjanov04:57
*** _SergeyLukjanov is now known as SergeyLukjanov04:57
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack
* zaro likes mordred's idea05:03
*** guohliu has quit IRC05:03
*** rcleere has quit IRC05:05
zarotime for some zzzzzzz..05:05
*** jamesmcarthur has quit IRC05:06
SergeyLukjanovclarkb, hi05:08
SergeyLukjanovclarkb, you've placed comment in
SergeyLukjanovto name gate-climate-devstack-dsvm05:08
SergeyLukjanovwhy devstack-dsvm if dsvm == devstack-vm?05:09
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack
clarkbbecause of the way zuul knows that a test is running on a single use VM. it is just a regex on the job name05:14
clarkbyou can see this in the layout.yaml file under the jobs list05:14
SergeyLukjanovname: ^.*(dsvm|devstack-vm).*$05:15
clarkbya that05:15
SergeyLukjanovit looks like it'll not accept devstack-dsvm05:15
clarkbit will .*dsvm.* matches05:15
*** mrodden has joined #openstack-infra05:15
*** rongze has joined #openstack-infra05:16
SergeyLukjanovoops, incorrect wording, I mean that -dsvm is enough and so why to name it "-devstack-devstack-vm"?05:16
*** yjiang5_1 has joined #openstack-infra05:17
*** guitarza1 has joined #openstack-infra05:17
clarkboh because it is running the devstack exercisea. should be gate-climate-devstack-dsvm05:17
*** gfa_ has joined #openstack-infra05:19
SergeyLukjanovclarkb, ok, got it, thanks05:19
*** rainya has quit IRC05:20
*** LinuxJedi_ has joined #openstack-infra05:20
StevenKclarkb: Was it you that mentioned the Git for 4 years and up talk yesterday? I watched the LCA 2013 recording of it05:21
*** michchap_ has joined #openstack-infra05:21
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack
clarkbStevenK: yes, Schwern is a friends roommate and I have seen the talk too. I find it really helpful understanding what git is doing05:21
*** marun has quit IRC05:22
*** mrodden1 has quit IRC05:22
*** guitarzan has quit IRC05:22
*** stevebaker has quit IRC05:22
*** sandywalsh_ has quit IRC05:22
*** yjiang5 has quit IRC05:22
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack-dsvm
*** michchap has quit IRC05:22
*** gfa has quit IRC05:22
*** LinuxJedi has quit IRC05:22
*** kpepple has quit IRC05:22
*** ttx has quit IRC05:22
*** pvo has quit IRC05:22
clarkbmakes understanding the differences between branches and tags and detached heads easy as well as rebase and other things05:22
*** kpepple has joined #openstack-infra05:22
*** antonym has quit IRC05:22
*** ttx has joined #openstack-infra05:22
StevenKI don't think I want to build a model of an openstack repo, though ...05:22
*** stevebaker has joined #openstack-infra05:22
*** marun has joined #openstack-infra05:22
*** pvo has joined #openstack-infra05:22
*** ttx has quit IRC05:22
*** ttx has joined #openstack-infra05:22
*** rainya has joined #openstack-infra05:22
*** rongze has quit IRC05:28
openstackgerritA change was merged to openstack-infra/devstack-gate: Use get-pip for pip installation
*** amotoki has quit IRC05:30
*** sandywalsh_ has joined #openstack-infra05:30
*** MarkAtwood has joined #openstack-infra05:36
openstackgerritA change was merged to openstack-dev/pbr: show diff between pip installed packages and requirements
*** FallenPegasus has quit IRC05:38
chandankumarwhen i send a review to openstack review.05:44
chandankumarwhen jenkins review it , i am not getting any mails in my mailbox.05:45
clarkbchandankumar: under your gerrit settings have you configured gerrit to send you email for things?05:45
clarkbalso double check your prefered email address is correct05:45
chandankumarclarkb, yes05:46
chandankumarclarkb, the email is configured there.05:47
chandankumarclarkb, it is correct.05:48
chandankumarclarkb, it is happening from that day when i use my laptop to send reviews to gerrit05:48
chandankumargenerally i use my desktop to send review05:48
chandankumarclarkb, i am not figuring out what is happening with it.05:50
clarkbchandankumar: do you have an example of a change that worked and a change that did not work?05:50
chandankumarclarkb, for thsi review
chandankumarwhen tom review my patch i have not got the mail05:52
*** nicedice has quit IRC05:53
clarkbI am still getting email so it isn't breaking everywhere. However I stopped sending gerrit mail to my gmail account05:55
clarkbchandankumar: have you ever received email from gerrit? I think you may need to confirm the address with gerrit (it has been so long I am trying to remember if that was requried)05:56
chandankumarclarkb, confirmation email i have not received from gerrit05:57
*** dstanek has quit IRC05:57
clarkblet me look in my email history to see if I got one of those05:57
*** rongze has joined #openstack-infra05:58
clarkbI got one when I added my second address but don't see one for the original05:59
clarkbchandankumar: preferred email is set correctly at ?05:59
*** dstanek has joined #openstack-infra06:00
chandankumarclarkb, it is setted there.06:00
chandankumarclarkb, no idea what is happening with my gerrit account.06:02
clarkbis it possible that the mail from is being filter?06:03
openstackgerritA change was merged to openstack-dev/pbr: Use wheels for installation
chandankumarclarkb, after 29 nov , i have not received mail from review@openstack.org06:04
*** SergeyLukjanov has quit IRC06:04
clarkbchandankumar: can you link to a change where you did receive email?06:04
*** SergeyLukjanov has joined #openstack-infra06:05
*** paul-- has quit IRC06:07
clarkbdid you get email for ?06:07
*** SergeyLukjanov has quit IRC06:07
*** paul-- has joined #openstack-infra06:10
clarkbmy ability to debug right now is limited (almost bed time) so trying to gather as much info as possible06:10
chandankumarclarkb, no i have not received06:13
clarkbchandankumar: did it ever work? if so please link to a change where it did work06:14
*** paul-- has quit IRC06:14
chandankumarclarkb, up to this patch Fix the wrong PATCH body of v2 for image update06:14
chandankumari have received the mails06:14
clarkbinteresting so you did receive email for a change submitted by someone else that you commented on06:16
clarkbbut not for changes submitted by you06:16
clarkbok good to know, I will try poking at it tomorrow06:17
chandankumarclarkb, Thank you06:18
*** dstanek has quit IRC06:20
*** kui has quit IRC06:21
*** fbo_away is now known as fbo06:26
*** fifieldt_ has quit IRC06:27
*** rongze has quit IRC06:30
*** rongze has joined #openstack-infra06:30
*** pcrews has quit IRC06:32
*** nosnos has quit IRC06:34
*** nosnos has joined #openstack-infra06:35
*** jcooley_ has joined #openstack-infra06:41
*** sarob has joined #openstack-infra06:42
*** fbo is now known as fbo_away06:50
*** che-arne_ has quit IRC06:51
*** jcooley_ has quit IRC06:51
*** boris-42 has joined #openstack-infra06:51
*** vkozhukalov has quit IRC06:51
*** Ryan_Lane has joined #openstack-infra06:51
*** nati_ueno has joined #openstack-infra06:52
*** SergeyLukjanov has joined #openstack-infra06:55
*** Ryan_Lane has quit IRC06:56
*** marun has quit IRC06:57
*** DinaBelova has joined #openstack-infra06:57
*** denis_makogon has joined #openstack-infra06:58
*** yolanda has joined #openstack-infra06:58
*** MarkAtwood has quit IRC07:02
*** wenlock has quit IRC07:03
*** nosnos_ has joined #openstack-infra07:05
*** guohliu has joined #openstack-infra07:06
*** nosnos has quit IRC07:08
*** nati_ueno has quit IRC07:10
*** antonym has joined #openstack-infra07:17
*** rwsu has quit IRC07:24
*** sarob has quit IRC07:25
*** sarob has joined #openstack-infra07:25
*** yongli has quit IRC07:28
*** sarob has quit IRC07:30
*** amotoki has joined #openstack-infra07:33
*** pblaho has joined #openstack-infra07:37
jog0clarkb: I see a glance pull loop07:38
jog0this *may* be related
jog0clarkb: here is teh bug
jog0so yes we havea bug for it07:46
jog0I think thast it at least07:47
jog0I think we have a nasty libvirt bug(s)07:48
*** SergeyLukjanov has quit IRC07:48
*** rwsu has joined #openstack-infra07:48
*** fifieldt has quit IRC07:49
*** Ryan_Lane has joined #openstack-infra07:51
*** cederb02 has joined #openstack-infra07:52
*** afazekas has quit IRC07:55
*** Ryan_Lane has quit IRC07:55
*** sarob has joined #openstack-infra07:56
*** flaper87|afk is now known as flaper8707:59
*** cederb02 has quit IRC08:01
*** avishayb has joined #openstack-infra08:08
*** marun has joined #openstack-infra08:09
*** cederb02 has joined #openstack-infra08:12
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1257626
uvirtbotLaunchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New]
*** cederb02 has quit IRC08:16
*** guohliu has quit IRC08:17
*** sarob has quit IRC08:18
*** nosnos_ has quit IRC08:19
openstackgerritSascha Peilicke proposed a change to openstack/requirements: Allow Django 1.5 and 1.6
*** nosnos has joined #openstack-infra08:19
*** inkerra has quit IRC08:20
*** metabro has quit IRC08:20
jog0clarkb: for bug 1257626, were there any changes to hpcloud recently?08:21
uvirtbotLaunchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New]
*** GheRivero has quit IRC08:22
jog0and how we use it, if so that may trigger this08:22
*** inkerra has joined #openstack-infra08:22
*** metabro has joined #openstack-infra08:22
*** Ryan_Lane has joined #openstack-infra08:29
*** markwash has quit IRC08:29
*** vkozhukalov has joined #openstack-infra08:30
*** markmcclain has joined #openstack-infra08:33
*** matsuhashi has quit IRC08:35
*** matsuhashi has joined #openstack-infra08:36
*** SergeyLukjanov has joined #openstack-infra08:38
*** boris-42 has quit IRC08:40
*** denis_makogon has quit IRC08:41
*** moted has quit IRC08:41
*** moted has joined #openstack-infra08:42
*** sileht has quit IRC08:43
*** sandywalsh_ has quit IRC08:44
*** sileht has joined #openstack-infra08:47
*** SergeyLukjanov has quit IRC08:49
*** nprivalova has joined #openstack-infra08:56
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1257641
uvirtbotLaunchpad bug 1257641 in tempest "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances" [Undecided,New]
*** sandywalsh_ has joined #openstack-infra08:57
*** jcoufal has joined #openstack-infra08:58
*** hashar has joined #openstack-infra09:00
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1257626
uvirtbotLaunchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New]
*** mkoderer_ has joined #openstack-infra09:05
*** zigo has joined #openstack-infra09:05
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1257641
uvirtbotLaunchpad bug 1257641 in tempest "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances" [Undecided,New]
*** branen__ has joined #openstack-infra09:05
*** primemin1sterp has joined #openstack-infra09:07
*** amotoki has quit IRC09:08
*** ritesh has joined #openstack-infra09:08
*** jpich has joined #openstack-infra09:08
*** fesp has joined #openstack-infra09:09
*** belliott_ has joined #openstack-infra09:11
*** SlickN1k has joined #openstack-infra09:11
*** branen_ has quit IRC09:12
*** zigo_ has quit IRC09:12
*** primeministerp has quit IRC09:12
*** mkoderer has quit IRC09:12
*** Alex_Gaynor has quit IRC09:12
*** flaper87 has quit IRC09:12
*** zaro has quit IRC09:12
*** clarkb has quit IRC09:12
*** belliott has quit IRC09:12
*** SlickNik has quit IRC09:12
*** che-arne has joined #openstack-infra09:12
*** fesp is now known as flaper8709:12
*** belliott_ is now known as belliott09:12
*** SlickN1k is now known as SlickNik09:12
*** mkoderer_ is now known as mkoderer09:12
*** markmcclain has quit IRC09:13
*** flaper87 has quit IRC09:13
*** flaper87 has joined #openstack-infra09:13
*** derekh has joined #openstack-infra09:15
*** yongli has joined #openstack-infra09:17
*** sarob has joined #openstack-infra09:17
*** zaro has joined #openstack-infra09:17
*** Alex_Gaynor has joined #openstack-infra09:18
*** clarkb has joined #openstack-infra09:19
*** SergeyLukjanov has joined #openstack-infra09:22
*** lianglin has joined #openstack-infra09:29
lianglin@fungi, do you know if community accept to run Integration in simulator or not?09:30
*** yassine has joined #openstack-infra09:30
*** sarob has quit IRC09:31
*** romcheg has quit IRC09:32
*** salv-orlando has joined #openstack-infra09:33
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge
*** afazekas has joined #openstack-infra09:36
lianglinwhere can I find the criterion of post CI test (including gate and third-party testing)? I would like to see what kind of test must be executed, can it run against simulator or not, voting or non-voting...09:36
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge
*** bingbu has quit IRC09:40
*** markmc has joined #openstack-infra09:42
*** boris-42 has joined #openstack-infra09:46
*** dizquierdo has joined #openstack-infra09:49
*** yaguang has joined #openstack-infra09:56
*** sarob has joined #openstack-infra09:56
*** salv-orlando_ has joined #openstack-infra10:01
*** nosnos_ has joined #openstack-infra10:03
*** salv-orlando has quit IRC10:03
*** salv-orlando_ is now known as salv-orlando10:03
*** jorisroovers has joined #openstack-infra10:03
*** nosnos has quit IRC10:06
*** Ryan_Lane has quit IRC10:09
openstackgerritValeriy Ponomaryov proposed a change to openstack-infra/config: Add gate-manila-devstack job
*** BobBallAway is now known as BobBall10:10
*** DinaBelova has quit IRC10:10
*** mkerrin has joined #openstack-infra10:13
*** avishayb has quit IRC10:18
*** matsuhashi has quit IRC10:22
*** nprivalova has quit IRC10:22
*** markmcclain has joined #openstack-infra10:22
*** lianglin has quit IRC10:28
*** sarob has quit IRC10:30
*** yaguang has quit IRC10:34
*** ritesh has quit IRC10:34
*** markmcclain has quit IRC10:35
*** johnthetubaguy has joined #openstack-infra10:35
*** paul-- has joined #openstack-infra10:35
*** markmcclain has joined #openstack-infra10:35
*** rossella_s has joined #openstack-infra10:35
*** nati_ueno has joined #openstack-infra10:37
*** nprivalova has joined #openstack-infra10:41
*** matsuhashi has joined #openstack-infra10:45
*** dizquierdo has quit IRC10:52
*** rongze has quit IRC10:53
*** DinaBelova has joined #openstack-infra10:55
*** sarob has joined #openstack-infra10:56
*** rongze has joined #openstack-infra11:11
*** SergeyLukjanov has quit IRC11:12
*** jorisroovers has quit IRC11:12
*** DinaBelova has quit IRC11:15
*** rfolco_ has joined #openstack-infra11:15
*** avishayb has joined #openstack-infra11:24
*** yassine has quit IRC11:25
*** sarob has quit IRC11:28
*** DinaBelova has joined #openstack-infra11:41
*** SergeyLukjanov has joined #openstack-infra11:43
*** jergerber has quit IRC11:43
*** rongze has quit IRC11:45
anteayalianglin: have you looked at this page?
*** dizquierdo has joined #openstack-infra11:49
*** rongze has joined #openstack-infra11:51
*** tma996 has joined #openstack-infra11:51
*** nprivalova has quit IRC11:55
*** sarob has joined #openstack-infra11:56
*** jorisroovers has joined #openstack-infra12:03
BobBallDo the check jobs set up a tempest.conf file, or just use the defaults from tempest?12:03
*** paul-- has quit IRC12:12
sdagueBobBall: they setup tempest via devstack12:17
sdaguedevstack will configure a tempest that tests the features you've turned on in it12:17
*** prad has joined #openstack-infra12:17
*** dims has joined #openstack-infra12:18
BobBallah - do you know where that script is?12:19
BobBallI want to make a test change and have something that'll be usabel in check jobs...12:19
BobBallbasically I want a new smaller flavor for tempest to use - it currently uses 512M instances in my setup and I don't know if devstack is already fixing that to create+use a smaller flavor - but cirros only needs 20M so I was going to give it 128M12:20
BobBallParallel tempest tests are failing for me because it wants to start too many VMs at once :)12:20
BobBalland I run out of memory12:20
sdagueBobBall: so we already do that12:22
sdaguelib/tempest - and there are tiny and nano there12:22
*** paul-- has joined #openstack-infra12:23
BobBallI'm not doing that in my CI so I wanted to set it up to do it :)12:23
BobBallthanks - I'll look there12:23
sdaguebasically you need to not define DEFAULT_INSTANCE_TYPE12:23
BobBallahhhhh perfect!!12:24
BobBallindeed we are specifying it12:24
BobBallI knew someone would have fixed that one already... ;)12:25
*** nprivalova has joined #openstack-infra12:28
*** sarob has quit IRC12:28
*** CaptTofu has quit IRC12:30
*** CaptTofu has joined #openstack-infra12:31
*** salv-orlando_ has joined #openstack-infra12:33
*** salv-orlando has quit IRC12:36
*** salv-orlando_ is now known as salv-orlando12:36
vkozhukalovIf I want to create project specific jobs (aka modules/openstack_project/files/jenkins_job_builder/config/project_name.yaml) where do I need to include this file to allow Jenkins Job Builder to know about this particular yaml file?12:39
openstackgerritA change was merged to openstack/requirements: Allow Django 1.5 and 1.6
openstackgerritA change was merged to openstack/requirements: Bump python-savannaclient version to >= 0.4.0
vkozhukalovor it includes all yaml files which present in that directory?12:40
*** SergeyLukjanov is now known as _SergeyLukjanov12:42
*** _SergeyLukjanov has quit IRC12:42
*** dolphm has joined #openstack-infra12:46
anteayavkozhukalov: hi there12:47
anteayacan I ask you to expand with an example?12:47
anteayawhat job are you trying to create?12:48
*** dstanek has joined #openstack-infra12:48
*** sandywalsh_ has quit IRC12:49
*** smarcet has joined #openstack-infra12:50
openstackgerritYuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror
*** jcoufal has quit IRC12:52
anteayavkozhukalov: here is the layout.yaml file that calls the individual jenkins jobs:
anteayais that what you were looking for?12:53
vkozhukalovI am trying to create functional testing job for ironic. I am going to make one VM install ironic on it then make another VM, then boot second VM via PXE from the first one. Ok. Actully it does not matter how it is supposed to work. I just try to understand how to integrate that test into -infra. I can see project specific job in tripleo.yaml and can not understand how jenkins job builder is supposed to know about this file.12:54
anteayalet me see if I can help12:56
*** sarob has joined #openstack-infra12:56
*** jcoufal has joined #openstack-infra12:57
anteayavkozhukalov: openstack-infra/config/modules/jenkins/files/slave_scripts/<script>.sh is picked up by openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/micros.yaml which is picked up by openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml which talks to13:00
anteayawhich is used by oenstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml13:01
anteayaa bit of a trail13:01
anteayatake some time with that13:01
anteayaand see if you have the path you need to do your work13:01
*** matsuhashi has quit IRC13:03
*** sandywalsh_ has joined #openstack-infra13:03
vkozhukalovanteaya, thanks. will go into it.13:03
*** dprince has joined #openstack-infra13:03
*** SergeyLukjanov has joined #openstack-infra13:05
*** nosnos_ has quit IRC13:07
*** nosnos has joined #openstack-infra13:08
*** yamahata_ has joined #openstack-infra13:09
*** paul-- has quit IRC13:09
*** nprivalova has quit IRC13:11
*** paul-- has joined #openstack-infra13:14
jog0clarkb: I think '    Add HPCloud region-a to nodepool.13:15
jog0Change-Id: I809303c5dd2f25f2b4dad638568fe4b8b105981813:15
jog0may be breaking the large-ops tests13:15
*** ruhe has joined #openstack-infra13:15
*** CaptTofu has quit IRC13:15
jog0this would be true if region-a is slower what we had a week ago13:15
*** CaptTofu has joined #openstack-infra13:16
*** dims has quit IRC13:17
*** dims has joined #openstack-infra13:17
*** sandywalsh_ has quit IRC13:20
*** thomasem has joined #openstack-infra13:23
*** DinaBelova has quit IRC13:24
anteayavkozhukalov: k, hope it helps13:28
*** sarob has quit IRC13:29
*** xchu has joined #openstack-infra13:29
*** yaguang has joined #openstack-infra13:30
*** fbo_away is now known as fbo13:31
*** nprivalova has joined #openstack-infra13:31
ArxCruzjeblair: around ?13:32
*** matsuhashi has joined #openstack-infra13:32
*** paul-- has quit IRC13:32
sdagueArxCruz: it's 5:30am his time :)13:32
sdagueso expect him up in a few more hours13:32
ArxCruzsdague: oh...13:33
ArxCruzwould you like me to create new patches with dependencies of this?13:33
*** herndon_ has joined #openstack-infra13:34
sdagueyes please13:34
*** DinaBelova has joined #openstack-infra13:35
*** mriedem has joined #openstack-infra13:35
*** sandywalsh_ has joined #openstack-infra13:36
*** markmcclain has quit IRC13:38
*** paul-- has joined #openstack-infra13:39
*** nosnos has quit IRC13:39
*** CaptTofu has quit IRC13:39
*** ruhe is now known as ruhe_13:41
openstackgerritArx Cruz proposed a change to openstack-infra/devstack-gate: Adding an option to use qpid instead of rabbit or zeromq
*** nprivalova has quit IRC13:43
*** ruhe_ is now known as ruhe13:44
*** jorisroovers has quit IRC13:48
*** jorisroovers has joined #openstack-infra13:48
*** jorisroovers has quit IRC13:50
*** jorisroovers has joined #openstack-infra13:52
openstackgerritRoman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic
sdaguejeblair: it would be nice to coordinate moving the elastic_recheck console tree today13:55
sdagueas there are a stack of fixes in there which would be handy to land13:55
sdagueand not dual maintain13:56
*** dkranz has joined #openstack-infra13:56
*** sarob has joined #openstack-infra13:56
*** markmcclain has joined #openstack-infra14:02
*** nprivalova has joined #openstack-infra14:02
*** paul-- has quit IRC14:05
*** julim has joined #openstack-infra14:05
*** bpokorny1 has joined #openstack-infra14:06
*** mfer has joined #openstack-infra14:06
*** yamahata_ has quit IRC14:07
*** paul-- has joined #openstack-infra14:07
*** yamahata_ has joined #openstack-infra14:07
openstackgerritRoman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic
*** heyongli has joined #openstack-infra14:11
*** markmcclain has quit IRC14:12
*** rongze has quit IRC14:14
*** mfer has quit IRC14:17
fungijog0: he and i were discussing it last night... the new hp regions are definitely slower, on par with rackspace's. the current suggestion is to switch them to check-only like we do with devstack nodes on rackspace14:17
jog0fungi: ahh yeah14:17
jog0that makes sense14:17
*** paul-- has quit IRC14:18
*** mfer has joined #openstack-infra14:18
jog0that is causing at least two bugs14:18
jog0and *maybe* the py27 nova issues14:18
*** rongze has joined #openstack-infra14:18
*** dstanek has quit IRC14:19
fungioh, too fun14:19
fungijog0: they wouldn't cause problems with nova unit testing, since they're only being used for integration jobs14:20
jog0fungi:  I just figured that out as well, was hoping to kill 3 birds with one stone I will have to settle for 214:21
jog0anyway I have to go AFK For a bit14:21
*** loq_mac has joined #openstack-infra14:23
*** loquacities has joined #openstack-infra14:29
*** nprivalova has quit IRC14:29
*** loq_mac has quit IRC14:30
*** sarob has quit IRC14:30
*** weshay has joined #openstack-infra14:31
*** loquacities has quit IRC14:35
*** dkliban has joined #openstack-infra14:42
*** guitarza1 is now known as guitarzan14:43
*** dkliban_ has joined #openstack-infra14:43
jog0fungi: for the large-ops issues, I am thinking we should drop the number of vms down to 100 (so a drop of 33%)14:45
jog0that should help the issue14:45
fungioh, so it's 150 right now?14:46
fungiyou were originally able to ferret out interesting scale issues at 50 anyway, right?14:46
jog0in part because that test originally  failed  for 50 instances and to fix things further in nova network is significant work (batching network commands etc)14:46
fungiso 100 still seems like a good safety margin to prevent those from returning14:46
*** dkliban has quit IRC14:47
jog0fungi: yup it was root wrap was slow14:47
jog0and we called it too much14:47
jog0I'll file a bug with neutron and nova and then drop the value to 10014:47
ruhe see error msg in the bottom. does it mean that partition ran out of space?14:48
mordredfungi: we don't have any graphs or collected numbers do we?14:48
*** pabelanger_ has joined #openstack-infra14:48
fungimordred: for the new region performance? i think we may have job run times in graphite but it would take some work to isolate and compare them against the old regions14:49
*** pabelanger_ has quit IRC14:49
*** pabelanger_ has joined #openstack-infra14:49
*** pabelanger has quit IRC14:50
mordredfungi: bleh. I hate work14:50
*** pabelanger_ is now known as pabelanger14:50
*** pabelanger_ has joined #openstack-infra14:50
vkozhukalovanteaya, ok. again about ironic functional testing. As far as i need to launch a couple of VMs and boot one of them via PXE, I found out how it is done in tripleo. They have shell code to deal with emulated bare metal (bm_poseurs), but this code in tripleo-incubator. There is the -infra/tripleo-ci project which uses code from tripleo-incubator to deal with bm_poseurs. It is supposed that in the future tripleo-ci will get emulated bare meta14:50
vkozhukalovl environments from via API. So the question is should I create -infra/ironic-ci project and do the things as in tripleo-ci and should I use code from tripleo-incubator to launch bm_poseurs or I need to put the bm_poseurs code into ironic?14:50
*** pabelanger has quit IRC14:51
openstackgerritArx Cruz proposed a change to openstack-infra/devstack-gate: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER
*** pabelanger has joined #openstack-infra14:53
fungivkozhukalov: i think those are questions you want to direct to #tripleo for the moment. the bare metal test cluster is not in production for upstream ci, and i'm unclear on their anticipated timeline14:53
*** riteshnanda has joined #openstack-infra14:53
fungivkozhukalov: what we do have available for now is the ability to start up a vm running devstack on it, and then have it manage small (generally cirros) qemu paravirt guests14:54
fungivkozhukalov: so if your testing can be limited to launching a couple of tiny cirros vms under qemu on a clound vm running devstack and have one of those (or the devstack context itself) provide the necessary pxe boot configuration for the cirros virtual machines, that might be something we could accommodate with the current implementation14:56
*** sarob has joined #openstack-infra14:56
jog0fungi: we have some interesting findings with py27 failures inthe tripleo room14:57
* fungi /joins14:57
jog0it looks like its a performance issue, TL;DR there is a 160 second timeout on all nova tests and the migration tests are alwas slow14:58
*** DinaBelova has quit IRC14:58
*** DennyZhang has joined #openstack-infra15:01
openstackgerritArx Cruz proposed a change to openstack-infra/config: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER in jjb
*** jamesmcarthur has joined #openstack-infra15:02
openstackgerritArx Cruz proposed a change to openstack-infra/config: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER in jjb
*** reaper has joined #openstack-infra15:03
vkozhukalovfungi, I've already asked them. I know that bare metal test cluster is not ready, and I can launch bm_poseurs via virsh CLI. It is ok. I'am wondering if it is possible to start two VMs and boot the first VM from the PXE configured on the second one. Or I need to run VM and then run another qemu VM inside VM?15:04
*** herndon_ has quit IRC15:05
vkozhukalovfungi, I mean if it is possible to launch two VMs on jenkins bare metal node.15:05
fungivkozhukalov: yes, right now we have a design limitation of one cloud vm per jenkins job. so inside that vm you can create more nested virtual machines, but those have to be paravirtualized because none of the donated cloud resources we use provide hardware support for nesting15:05
vkozhukalovfungi, ok I see.15:06
fungivkozhukalov: i think the tripleo plan is to have a jenkins job correspond to a control node in the bare metal cluster instead, and it would then have the ability to launch virtual (or even physical) machines as needed for that particular job within an isolated tenant environment15:06
fungibut they can probably explain the details far better than i15:07
*** thedodd has joined #openstack-infra15:07
*** markwash has joined #openstack-infra15:09
*** oubiwann_ has joined #openstack-infra15:10
*** markmcclain has joined #openstack-infra15:11
*** yaguang has quit IRC15:12
*** dstanek has joined #openstack-infra15:16
*** SergeyLukjanov is now known as _SergeyLukjanov15:16
*** _SergeyLukjanov has quit IRC15:16
*** riteshnanda has quit IRC15:16
*** ywu has quit IRC15:16
*** denis_makogon has joined #openstack-infra15:19
*** heyongli has quit IRC15:19
anteayawe are sitting here workong on neutron-grenade testing, EmilienM has a patch up here: that hopefully addresses jeblair's latest comments, does anyone have time for a review?15:20
anteayacurrently that patch is a block for our work15:20
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Drop TEMPEST_LARGE_OPS_NUMBER by 33% to 100
jog0fungi: ^ theres the patch15:22
jog0ohh and I got a nice round number for my gerrit number too15:22
pleia2woo 60k15:23
*** matsuhashi has quit IRC15:23
* jog0 wonders why gerrit doesn't use hex numbers15:23
fungianteaya: technically i don't think it addresses jeblair's comment on the previous patchset, but it does seem to reflect the irc discussion EmilienM had with clarkb after that15:25
*** riteshnanda has joined #openstack-infra15:26
anteayafungi: oh okay great15:27
*** xchu has quit IRC15:27
fungisdague: dtroyer_zz: any idea whether adding (voting) grenade-neutron jobs to grenade and not to neutron is the right path forward there? i would have expected it to be applied to neutron only or both neutron and grenade, but not just grenade...15:27
*** esker has joined #openstack-infra15:28
anteayafungi: EmilienM wants to know if he also needs to address periodic jobs due to this change15:28
fungisdague: dtroyer_zz:
*** sarob has quit IRC15:28
fungianteaya: i think clarkb said yesterday that he was taking a wait-and-see approach on grenade jobs before going into a refactoring similar to what he did with the devstack/tempest jobs15:29
*** senk has joined #openstack-infra15:29
openstackgerritJoe Gordon proposed a change to openstack-infra/devstack-gate: Drop TEMPEST_LARGE_OPS_NUMBER by 33% to 100
anteayafungi: so we are waiting to hear back from clarkb?15:29
sdaguefungi: actually, it should be both15:29
fungisdague: thanks. i thought so. i'll -1 it with pointers15:30
sdaguealso, probably worth starting it as non-voting just to get the result stream15:30
*** jcooley_ has joined #openstack-infra15:30
fungiyou read my mind15:30
*** oubiwann_ has quit IRC15:31
*** rfolco has quit IRC15:32
*** dcramer_ has joined #openstack-infra15:33
*** wenlock has joined #openstack-infra15:34
*** datsun180b has joined #openstack-infra15:34
mordredjog0: nice!15:34
*** nprivalova has joined #openstack-infra15:34
jog0fungi: here is the bug for nova and neutron
uvirtbotLaunchpad bug 1257788 in nova "nova-network doesn't handle large amount of concurrent booting instances well" [Low,New]15:35
*** markwash has quit IRC15:35
*** markwash has joined #openstack-infra15:38
*** DennyZhang has quit IRC15:40
*** ruhe is now known as ruhe_15:41
*** ruhe_ is now known as ruhe15:42
*** SergeyLukjanov has joined #openstack-infra15:44
openstackgerritChuck Short proposed a change to openstack-dev/pbr: python3: Fix python3.2 installation
ttxmordred: hey, was wondering if I needed to do anything for oso.rootwrap to start showing up on PyPI once tagged15:46
*** rongze has quit IRC15:46
ttxmordred: i'm about to push a 1.0.0 tag there15:46
fungittx: i can check15:46
ttxfungi: thx!15:47
fungittx: returns "Not Found ()" so the project presumably needs to be registered15:47
ttxfungi: yeah, was wonderig if that happened automagically15:47
ttxfungi: let me know if I need to do anything, but i suspect not ?15:48
jeblairi'll do it real quick15:48
mordredit does not yet happen automagically15:48
mordredoh! look. it's jeblair15:48
* mordred really wants to add that to manage-projects15:48
ttxmordred: I'm now expecting automation by default15:49
ttxmy script that does MP branch creation, waits for the tarball build and tags all bugs fixreleased sort of works, btw15:49
ttxjeblair: so I should be able to tag 1.0.0 and magic will happen, right ?15:50
*** elasticio has joined #openstack-infra15:50
zulis there a newer oslo.messaging on pypi?15:50
jeblairttx: for future reference, i create the project manually using the web form, only filling out name and version (0.0), then add 'openstackci' as an owner15:50
fungipermissions seem to be in place, i'll check to make sure we're getting properly-versioned tarballs on prereleases15:50
ttxjeblair: ack15:50
jeblairttx: yes, a tag should start the machinery now15:51
*** syerrapragada1 has joined #openstack-infra15:51
fungiwell, we seem to be getting master branch tarballs at least, so your sdist is exercised15:51
*** nprivalova has quit IRC15:51
fungii guess this will be the first ever tag, so no prerelease tarballs to check there15:51
jog0so Django==1.6 just wedged the gate
ttxfungi, jeblair: tag on its way15:52
fungittx: looks like it's in teh status page now15:52
fungitarballs.o.o looks right15:53
*** vkozhukalov has quit IRC15:53
fungipypi looks right for the new release now15:53
fungimirror rebuilds are happening, so you should be all set15:53
*** dolphm has quit IRC15:54
*** syerrapragada has quit IRC15:54
*** rcleere has joined #openstack-infra15:54
*** elasticio has quit IRC15:54
*** elasticio has joined #openstack-infra15:54
*** oubiwann_ has joined #openstack-infra15:56
*** sarob has joined #openstack-infra15:56
*** tma996 has quit IRC15:57
anteayafungi: awesome, thanks for the comments on 5903915:57
*** jroovers has joined #openstack-infra15:58
*** tma996 has joined #openstack-infra15:58
*** jorisroovers has quit IRC15:58
*** elasticio has quit IRC15:58
*** elasticio has joined #openstack-infra15:59
*** tma996 has quit IRC15:59
jeblairmordred: would like your help...16:00
jeblairmordred: i want to figure out why passed16:00
*** tma996 has joined #openstack-infra16:00
jeblairmordred: fungi is working on reverting that change, because in fact after it landed and the mirrors rebuilt, things started failing16:01
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Create and upload wheels
*** senk has quit IRC16:01
*** tma996 has quit IRC16:01
*** senk has joined #openstack-infra16:01
mordredjeblair: ok. looking16:02
*** rongze has joined #openstack-infra16:02
*** tma996 has joined #openstack-infra16:02
*** jroovers has quit IRC16:02
*** Abhishek_ has joined #openstack-infra16:02
mordredjeblair: it's the situation we were talking about yesterday16:03
mordredexcept a version of it we didn't consider16:03
mordredI think16:03
mordredhrm. unless we _used_ the mirror when we werent' supposed to16:03
jeblairmordred: it passed check, but shouldn't have16:04
mordredyeah. with you16:04
*** tma996 has quit IRC16:04
jeblairmordred: + '[' 0 -eq 0 -o openstack/requirements '!=' openstack/requirements ']'16:04
*** tma996 has joined #openstack-infra16:04
*** tma996 has quit IRC16:04
jeblairmordred: so that first clause is "$DEVSTACK_GATE_REQS_INTEGRATION" -eq "0"16:05
*** Abhishe__ has joined #openstack-infra16:05
*** tma996 has joined #openstack-infra16:05
jeblairmordred: i think that always means that we use the mirror unless we're doing the requirements integration test16:06
mordredjeblair: I believe you are correct. which, it what fungi was saying as a feedback to the change to using select-mirror16:06
mordredand then I countered that that's not what we want16:06
mordredwhich we eventually came around to agreeing on16:07
jeblairmordred: so perhaps the logic is flawed, because what we really want is: use the mirror unless we're doing the requirements integration test _or_ this is a change to the requirements project16:07
*** dstanek has quit IRC16:07
mordredI agree with your recent statement16:07
*** dstanek has joined #openstack-infra16:07
jeblairmordred: ok, what's the state of that change?16:07
*** Abhishek_ has quit IRC16:08
mordredjeblair: I need to fix then it can be re-reviewed16:08
jeblairmordred:  looks like it has the same conditional16:09
anteayaonce the mirrors have been repopulated with django 1.5.5 again, can a core pull the lever on this add ironic puppet module to stackforge? 7 upvotes and a happy jenkins:
anteayafyi, I'm sitting in the eNovance office :D16:09
mordredjeblair: yah - but once the other two changes are in, the conditional is largely irrelevant, because the selection will key off of openstack/reuqirements not being in the projects.txt file16:10
ttxanteaya: most people would assume Montreal at that point.16:10
*** tma996 has quit IRC16:10
anteayattx good point16:10
anteayaParis then for clarity16:10
*** tma996 has joined #openstack-infra16:10
*** tma996 has quit IRC16:11
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6"
*** tma996 has joined #openstack-infra16:11
*** Abhishe__ has quit IRC16:11
*** avishayb has quit IRC16:11
*** nprivalova has joined #openstack-infra16:11
fungimarkmcclain: thanks16:12
*** osanchez has joined #openstack-infra16:12
*** tma996 has quit IRC16:12
*** tma996 has joined #openstack-infra16:13
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Use requirements project list for mirror selection
mordredjeblair: ok. there's ^^ the pre-req16:15
jeblairmordred: okay, i agree that the patch series should fix this (i also agree with fungi that removing the '-o "$ZUUL_PROJECT" != "openstack/requirements"' bit would be nice)16:15
*** jcoufal has quit IRC16:16
mordredjeblair: shoudl I just go ahead and do that in the final patch?16:16
jeblairmordred: yeah, i think so.  it's so confusing to have that there but not mean anything.16:17
*** tma996 has quit IRC16:17
fungii believe the original logic there was that we should use upstream pypi for all requirements-integration jobs and also for all changes to the openstack/requirements project16:17
jeblairfungi: yeah, but the boolean logic is wrong, so it's just wrong currently, and a no-op in the new system.  but it _looks_ really important.  :)16:18
mordredjeblair: do we even need DEVSTACK_GATE_REQS_INTEGRATION anymore?16:18
mordredjeblair: if we havea genearlized trigger that means that all changes to ... oh, wait, we need to block it for that for pbr chagnes too16:18
mordredyeah. we nede it16:18
*** markmcclain has quit IRC16:19
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Use select-mirror in devstack-gate
mordredokie. there it is updated16:19
jeblairmordred: i don't think we need it16:19
jeblairmordred: grepping in infra/config says it's only used in the {pipeline}-requirements-integration-dsvm job, which is not in zuul's layout16:20
mordredjeblair: we do- we also run requirements integration on changes to pypi-mirror and pbr16:20
mordredoh. hrm. ok16:20
mordredwhat's the job we use in pbr, pypi-mirror and requirements to do that checking?16:20
mordredshouldn't that have the DEVSTACK_GATE_REQS_INTEGRATION override?16:21
*** sarob has quit IRC16:21
jeblairmordred: oh, wait, it is in layout.yaml16:21
fungii wonder if it got lost in the d-g refactor16:21
jeblairsorry. i was wrong, it's still used and required.16:21
jeblairand i learned something about git grep.16:22
*** oubiwann_ has quit IRC16:22
jeblairmordred: so i just checked a random devstack-precise slave, and there is no /opt/requirements repo16:22
fungijeblair: did you learn that it takes your cwd into account for tree recursion?16:22
mikalIs there a way to inject jobs into zuul without talking to it via gerrit?16:23
mordredjeblair: that's right - we haven't landed that patch yet16:23
jeblairfungi: no i learned it (understandably) doesn't follow symlinks.  :)16:23
*** danger_fo_away is now known as dangers16:23
fungioh, ha16:23
mikali.e. it would be nice to be able to force a run of a pipeline without having to add gerrit comments16:23
jeblairmikal: that's admin-only though16:23
mikaljeblair: OMG< that's exactly what I want16:24
mikaljeblair: I'm an admin for turbo hipster though, so that would work for me16:24
*** oubiwann_ has joined #openstack-infra16:24
fungimikal: the more recent trigger reworking in zuul is intended to make it possible to have other trigger types besides gerrit though, eventually16:24
fungioh, and aside form periodic, which was what got added when that happened16:24
jeblairmordred: aprvd16:25
mordredjeblair: awesome16:25
*** nprivalova has quit IRC16:25
fungii think i hadn't approved it because it depends on another change16:26
mordredit does16:26
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Fix pep8 tests
jeblairmordred: is that ready to land?16:27
fungiclarkb: had a list of the wheels-in-mirror changes with the order you wanted them merged, so was trying to follow that sequence though we may have lost track of the intended order for a couple since16:27
mordredjeblair: yes. it should be perfectly fine to land that. it will add the wheel mirrors to the list of things searched16:27
*** ryanpetrello has joined #openstack-infra16:27
mordredif a run does not have wheels enabled, that will be a no-op url scan16:27
jeblairfungi: are we okay on the lsb command thing?16:28
jeblairfungi: more words: do all our current slave platforms have the lsb_release command?16:28
mordredjeblair: we could rebase them off of each other though - they are not actually dependent - it was just a dependent workstream for me16:28
fungijeblair: yeah, looks like it. the fedora-devstack people can just add the necessary additional package to the rpms list to make that possible i think16:28
*** jamesmcarthur has quit IRC16:28
ryanpetrelloany chance I can convince somebody to take a look at ?=16:28
ryanpetrellopretty straightforward review16:28
ryanpetrellojust hoping to get the ceilometer tests passing so I can make a pecan release :)16:29
ryanpetrellowhoa, that was fast16:29
ryanpetrellooh, wait, wrong email16:29
mordredbut I'd love to make having lsb_release a pre-req for our stuff working, since, you know, LSB16:29
*** jamesmcarthur has joined #openstack-infra16:30
*** reaper has quit IRC16:30
fungiryanpetrello: out of curiosity, why does that job need to run on centos6 instead of ubuntu lts?16:30
openstackgerritA change was merged to openstack-infra/config: Add support for per-distro wheel mirrors
jeblairmordred: what's up with rax performance nodes?16:30
openstackgerritA change was merged to openstack-infra/config: Install requirements repo on slaves
ryanpetrellofungi: it's running ceilometer tests that interact w/ mongodb16:30
fungiryanpetrello: ahh, right! that old saw16:31
*** ^d has joined #openstack-infra16:31
*** ^d has joined #openstack-infra16:31
*** markwash_ has joined #openstack-infra16:31
fungione of the few packages which centos has a newer version of than ubuntu16:31
jeblairnever thought i'd see the day16:31
*** markwash has quit IRC16:32
*** markwash_ is now known as markwash16:32
mordredjeblair: uhm. I thought I had a patch up for that16:32
mordredjeblair: but it might be simply that I did the nodepool patch to allow that patch to be made16:33
BobBallCan I make tox abort on the first failure?16:33
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge
mordredjeblair: which I think is landed now16:33
mordredjeblair: so now we just need to add some nodepool entries with some flavor name filters16:34
ryanpetrellofungi: thanks :)16:34
fungiryanpetrello: if you want to see newer mongodb on ubuntu precise, bug 1228977 could use some attention16:35
uvirtbotLaunchpad bug 1228977 in nova "n-cpu seems to crash when running with libvirt 1.0.6 from ubuntu cloud archive" [High,Confirmed]
*** UtahDave has joined #openstack-infra16:35
openstackgerritA change was merged to openstack-infra/config: Run pecan-ceilometer gate tests on centos6
*** senk has quit IRC16:38
mikaljeblair: for that client, what would a valid trigger look like? Is it just the trigger name?16:38
mikali.e "zuul -v -c /etc/zuul/zuul.conf enqueue --trigger gerrit --pipeline check --project openstack/nova --change 59856 --patchset 1" doesn't seem to work for me.16:39
*** CaptTofu has joined #openstack-infra16:39
mikaljeblair: running that ends with the client saying "Waiting for job completion", but not entries in debug.log or zuul.log relating to a job being created16:40
russellbanyone want to play "spot the syntax error" with my hook function?  This fails on the line defining the function: ... This one works:
*** jgrimm has joined #openstack-infra16:41
jeblairmikal: is the running server from that patchset too?  (it adds both the client and necessary server support)16:41
mikaljeblair: ahh, no. I didn't restart zuul.16:42
*** adam_g has joined #openstack-infra16:42
*** adam_g has quit IRC16:42
*** adam_g has joined #openstack-infra16:42
* mikal kicks self16:42
*** jcooley_ has quit IRC16:43
openstackgerritQianLin proposed a change to openstack-infra/nodepool: Fix the issue which raise FloatingIpPoolNotFound
*** sHellUx has joined #openstack-infra16:44
*** Ryan_Lane has joined #openstack-infra16:46
*** UtahDave has quit IRC16:47
mikaljeblair: the other gotcha was the new zuul_url config parameter16:47
jeblairmikal: yes, i'm a bad upstream and make that a breaking change.  i need to add that to NEWS.16:48
jeblairmikal: good news is we're about to cut 2.0 and i'll try to stop doing that.  :)16:48
*** michchap_ has quit IRC16:48
*** dstanek has quit IRC16:49
mikaljeblair: oh, I don't care. I'd much rather you move quickly16:49
*** marun has quit IRC16:50
jeblairmikal: usually it's not too much of a burden to be backwards compat and move quickly (especially since we CD it)16:50
mikaljeblair: is it expected that a zuul server restart would require a restart of all the workers? Because that's what happened to me...16:52
zarogood morning16:53
jeblairmikal: no, they should automatically reconnect.  but we don't use turbo hipster :(16:53
*** ArxCruz has quit IRC16:53
*** ArxCruz has joined #openstack-infra16:54
mikaljeblair: huh, this might be a turbo hipster bug then16:54
mikaljeblair: I will hassle jhesketh about it16:54
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Remove braces from bash vars in solum job config
*** pcrews has joined #openstack-infra16:55
*** sarob has joined #openstack-infra16:56
fungirussellb: you might have also been able to fix that by doubling the ${{braces}} to escape them from the yaml parser16:57
* fungi can't remember when that is and isn't necessary16:57
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Document zuul_url breaking change
russellbfungi: ah, makes sense16:58
zarowould any infra-core have time to create a gerrit 2.8 branch so we can get the gerrit upgrade started?16:59
*** LinuxJedi_ is now known as LinuxJedi16:59
fungizaro: i'll take a look16:59
zarofungi: thnx17:00
fungizaro: we've got it tracking upstream/stable-2.8 right now... you want an openstack/2.8.0 branch spun off from that?17:00
jeblairzaro: i thought you were going to install a war from master?17:00
*** sarob has quit IRC17:00
jeblairupstream master17:00
*** oubiwann_ has quit IRC17:01
jeblairzaro: or are you working on both things in parallel?17:01
zarojeblair: i was thinking we want the change owner patch added on top, so we need the branch.17:01
*** dcramer_ has quit IRC17:02
*** fbo is now known as fbo_away17:03
fungifrom a timeline perspective, if it turns out to take almost as long to set up the new build machinery for buck and 2.8 as it takes them to release 2.9, we may get better gains focusing on the upgrade path part and less on the additional features17:03
zarojeblair: i initially thought install from master was better idea, but since we'll probably want to make some (UI) tweeks anyway so might as well go with our own fork.17:03
zaroyeah, it will be easier and faster to go with upstream master.17:04
zaroso if you prefer upstream master then maybe the 2.8 branch can wait.17:04
fungifor ui tweaks, if it's something which requires rebuilding teh warfile, we probably want to upstream patches to make those externally-configurable instead and wait for it to be in the following upgrade17:05
*** rongze has quit IRC17:05
jeblairfungi: that's preferable, but it could slow things down considerably; i think i like the idea of doing both in parallel...17:06
* zaro agrees, lets do in parallel.17:06
*** rongze has joined #openstack-infra17:07
*** sHellUx has quit IRC17:07
jeblairzaro: that way if you hit something that slows you down while working on master on review-dev, you can switch to getting the build machinery going17:07
*** oubiwann_ has joined #openstack-infra17:07
jeblairand we'll be ahead of the game if we need to use it17:07
fungicarrying patches to adjust the ui in ways which are accepted upstream simply gets us those changes a little faster at the expense of maintaining build infrastructure/tooling for gerrit, while obviously carrying patches which won't be accepted upstream gets us back to the "bad old days" of having a perpetual fork17:07
jeblairfungi: yeah, definitely don't want a perpetual fork17:07
fungibut if the effort to maintain build machinery is relatively minimal, then parallel effort seems fine to me17:07
jeblairi'd say focus on upstream master, and secondarily, hack on build machinery so it's there if we need it17:08
*** yamahata_ has quit IRC17:08
*** jcooley_ has joined #openstack-infra17:08
zarofungi: so just go ahead and branch with latest upstream 2.8?17:09
*** SergeyLukjanov is now known as _SergeyLukjanov17:09
fungizaro: call it openstack/2.8.0?17:09
* fungi checks to see what's been tagged on the upstream branch17:10
*** jpich has quit IRC17:10
fungilooks like the tag is just v2.8 (no micro version)17:10
*** loq_mac has joined #openstack-infra17:11
fungi12 hours ago17:11
zarofungi: gerrit channel says watch out for LDAP on upgrade.  we not using that are we?17:11
fungizaro: no ldap for us (yet anyway)17:11
fungizaro: there is now an openstack/2.8 branch on the openstack-infra/gerrit project17:12
BobBallHow do I find out the cinder config in check jobs?17:12
zarogreat, thnx.17:12
fungiBobBall: in which check jobs?17:12
BobBallfull tempest17:12
fungiBobBall: look at the logs linked for a completed run and the localrc used for devstack should be included17:13
BobBalloh, of course... duh...17:13
fungiunder the "logs" subdirectory there will be a localrc.gz or similar17:13
*** dcramer_ has joined #openstack-infra17:14
fungizaro: looks like there are even a couple of post-2.8 patches on that stable branch already17:14
fungiminor documentation fixes, it seems17:15
*** jcooley_ has quit IRC17:15
*** dolphm has joined #openstack-infra17:17
*** dstanek has joined #openstack-infra17:17
zarofungi: interesting upstream gerrit site is on 2.8-rc3 but i don't see a tag for rc2.17:18
fungizaro: yeah, i see tags for v2.8-rc0, v2.8-rc1 and v2.817:20
fungizaro: maybe they incremented rc versions without corresponding tags, or deleted tags from the repo17:20
*** loq_mac has quit IRC17:21
*** ruhe is now known as ruhe_17:21
*** loq_mac has joined #openstack-infra17:21
*** ruhe_ is now known as ruhe17:21
*** mrodden has quit IRC17:22
zarofungi: would not surprise me.17:22
fungithe subway map for the gate is amusing at the moment17:24
jeblairfungi: lots of trains to nowhere17:24
fungiwould make for a great twilight zone plot17:25
fungi"lots of trains to nowhere"17:25
*** pblaho has quit IRC17:27
ryanpetrello(╯°□°)╯︵ ┻━┻17:27
mikaljeblair: looking at the zuul report formatter, its not clear to me how you get the nice HTML style reports in gerrit. Our email reports are just the url. How are you turning that into a HTML link?17:28
fungiclearly my terminal font is lacking a few glyphs there17:28
*** jasond has joined #openstack-infra17:28
jeblairmikal: gerrit commentlink magic17:28
ryanpetrellofungi: don't mind me, just flipping a table over17:29
mikaljeblair: who the what now?17:29
ryanpetrelloso running the ceilometer stuff on centos doesn't seem to have solved it either17:29
ryanpetrellodifferent failure now, though17:29
jeblairmikal: so our reports are exacly as you are seeing, but gerrit does magic on them when displaying them17:29
mikaljeblair: oh, I see (a quick google helped!)17:29
mikalSo, I need to trick you guys into letting me tweak the gerrit config as well?17:29
jeblairmikal: you want your comments to be linkified in openstack gerrit when you post them?17:30
jeblairmikal: if so, you can just make your zuul comment syntax match what we're using, and it'll hit the same commentlink patterns17:30
mikaljeblair: yeah, that's what I want17:31
* mikal looks at that template17:31
fungimikal: the patterns we match are listed at
jasondhi, i'm trying to understand why the gate failed with "Build timed out" in  i've been looking at what appears to be a debug log (, but i'm not sure where to look.  the tests for this patch pass locally.  can someone advise me where to look next?17:31
*** datsun180b is now known as datsun_F40PH17:31
*** hashar has quit IRC17:32
mikaljeblair: so it looks to me based on that config that your results are in the form of a href?17:32
*** chandankumar has quit IRC17:33
jeblairmikal: i think if you set job_name_in_report=true in zuul.conf you'll get the right pattern17:33
mikaljeblair: oh, cool. I shall try that17:33
*** mrodden has joined #openstack-infra17:34
*** loq_mac has quit IRC17:36
*** loq_mac has joined #openstack-infra17:36
mikalOMG, I want an IRC reporter in zuul17:38
mikalThat would be funky17:38
*** senk has joined #openstack-infra17:38
*** vkozhukalov has joined #openstack-infra17:39
jeblairmikal: :)17:39
mikaljeblair: heh, be gentle. This is my first day poking around zuul.17:40
*** loq_mac has quit IRC17:41
mordredmikal: you're a zuul17:41
zarojeblair, fungi: an email just came out on gerrit mailing list. final 2.8 was just release today so i believe the v2.8 tag is the final.17:41
jeblairmikal: i'd accept that patch.  also, keep in mind you could also write an irc trigger.  :)17:41
* jeblair will be amused if zuul's UI ends up being everything _except_ a web interface.17:42
fungijasond: i believe that's the raw subunit stream returned via testr during that job run17:42
clarkbjog0: yes region-* is breaking things, I have a change up to only use region-* on check tests17:42
jeblairclarkb: because it's slow?17:43
*** elasticio has quit IRC17:43
jasondfungi: oh ok, is it meant to help with debugging?17:43
fungizaro: right, they tagged v2.8 ~13 hours ago, so sayeth teh git17:43
clarkbanteaya: correct I don't think grenade has established a job pattern yet so I don't want to require things in that space yet. Better to learn and see what is going on there17:43
clarkbjeblair: yes17:43
*** senk has quit IRC17:43
mordredfungi, zaro: that still means that 2.8 final doesn't have the zaro patch though, right?17:43
mikaljeblair: I can see a use case for an IRC trigger as well17:44
mikaljeblair: not that I will ever have the free time to implement such a thing17:44
fungijasond: i think you may be able to feed it into a testtools something or other, someone else here may have a more accurate suggestion there ;)17:44
fungimordred: i believe we don't expect that patch until 2.917:44
clarkbjeblair: fungi: mordred:
mordredfungi: ok. cool17:45
mordredclarkb: looking17:45
zaromordred: that's correct, no zaro patch.  that will be in 2.917:45
fungigah! failed gating on check-requirements-integration-dsvm17:45
mordredfungi: how did it fail there?17:46
mordredfungi: uhm. why is it pulling in d2to117:47
fungilooks like maybe oslo.messaging installation broke?17:47
clarkbjeblair: when I babysat the changes tests were taking longer but not too much longer 14 vs 18 minutes on swift functional and ~30 vs ~40 minutes for neutron tempest. But full tempest seems to do much worse17:49
*** oubiwann_ has quit IRC17:49
*** derekh has quit IRC17:50
jeblairclarkb: what were the full tempest times?  i'd extrapolate 45 -> 60 mins based on those 2 data points (and the assumption that full tempest runs take ~45 mins currently)17:52
*** rossella_s has quit IRC17:52
mordredfungi: W T F?17:52
clarkbI saw one test run into the 90 minute timeout just as it was running the last cli tests. So I think just above 90 minutes is the worst case, but would need to look at more test runs to see if that is an outlier17:53
clarkbjeblair: ^17:53
jeblairwow.  that's definitely not linear.17:53
*** oubiwann_ has joined #openstack-infra17:53
fungimordred: i'm trying to suss out if there's any obvious reference to d2to1 in either master or latest tag of oslo.messaging, but no17:54
*** SergeyLukjanov has joined #openstack-infra17:55
*** gyee has joined #openstack-infra17:55
mordredfungi: right. I also don't understand how that's getting in there, or what's going on in that moment where it's trying to install oslo.messaging and winding up re-installing pbr and finding d2to117:55
*** ndipanov has joined #openstack-infra17:55
*** michchap has joined #openstack-infra17:56
fungimordred: did oslo.messaging 1.2.0 ever get released? tags seem to jump from 1.2.0a11 to 1.3.0a1 and the d2to1 setup_requires was removed between those17:57
mordredfungi: I do not belive it did17:57
mordredmarkmc: ^^17:58
*** riteshnanda has quit IRC17:58
*** SergeyLukjanov has quit IRC17:58
*** che-arne has quit IRC17:59
*** senk has joined #openstack-infra17:59
*** dcramer_ has quit IRC17:59
fungiyeah, tarballs and pypi seem to confirm that as well17:59
*** dizquierdo has quit IRC18:01
*** oubiwann_ has quit IRC18:02
*** michchap has quit IRC18:03
fungimordred: seems it built oslo.messaging==1.3.0a2.5.g566c40e at so presumably that's what it's installing again later for glance18:04
ndipanovhey guys - anyone have any idea who owns the mox3 package on pypi, and more importantly - where's the code kept?18:04
*** senk has quit IRC18:04
mordredndipanov: I do!18:04
*** reed has joined #openstack-infra18:05
ndipanovmordred, awesome, thanks!18:05
mordredndipanov: - although I tohught zul and I moved it into gerrit18:06
mordredzul: ^^ ?18:06
*** wenlock has quit IRC18:06
ndipanovmordred, ok that clears a lot of things up - it's not in gerrit btw...18:06
clarkbI thought we werent goign to put effort into mox3 because mock'18:07
ndipanovclarkb, porting for example nova to mock as a project is a huge waste of time imho18:07
zulmordred:  i dont think so18:07
clarkbndipanov: so is forking a project18:07
ndipanovclarkb, true - so I want to figure out what we should do18:08
clarkbat the summit I thought we decided new tests shoudl use mock, old tests will be converted over time18:08
clarkbzul: ^ ?18:08
zulthats right18:08
ndipanovclarkb, that works fine if we can move to mox3 - which no one seems to be doing18:08
zulndipanov: keystoneclient has already switched over and there is a patch for oslo-incubator to start switching over18:10
zulwe are focused on the clients right now18:10
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Nodepool: cache all gerrit repos
ndipanovzul, awesome - there were some concerns about some bug-fixes/divergence with upstream mox18:11
ndipanovif you could call it that18:11
*** ruhe has quit IRC18:11
ndipanovzul, it's be awesome if you would weigh in on the thread in dev right now18:11
ndipanovit would *18:11
zulndipanov:  im aware but upstream is basically dead right now18:11
zulndipanov:  on the ml?18:11
ndipanovzul, yeah18:11
zulndipanov:  ok18:11
ndipanovzul, any plans to backport stuff?18:12
zulndipanov:  i thought there was a thread last month ;)18:12
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add zuul pack-refs cron
ndipanovthat's the same thread :)18:12
ndipanovzul, I just dug it up since I had to -2 a patch converting stuff to mock18:12
ndipanovzul, and there does not seem to be a clear move forward at least in nova18:13
clarkbright I think we are rehashing old discussions18:13
ndipanovclarkb, definitely we are18:13
clarkbthe clear move forward is start using mock today for new tests, work on converting old tests as part of the move to support python318:13
ndipanovclarkb, and... mova to mox3?18:14
clarkbat the summit there wasnt any desire to support mox318:14
fungiprojects currently using mox and aiming for py3k compatibility will need to rewrite those tests to use mock instead of mox before they can reach that goal18:14
*** wenlock has joined #openstack-infra18:15
ndipanovI see18:15
ndipanovso mox3 is dead as far as you guys are concerned18:16
fungiafaik mox3 was an experimental fork of a dead project18:16
fungiwhile mock is actively maintained and works with both python 2.x and 3.x18:17
clarkbmimic was the other option18:18
zulndipanov:  no mox is dead imho18:18
jeblairfungi, mordred: what's up with requirements and ?18:18
zulndipanov:  i just commented on the list18:18
mordredjeblair: we're trying to figure it out18:18
mordredthe failure there is WEIRD18:18
clarkbfungi: jeblair: any opposition to my approving ?18:18
fungijeblair: i'm still trying to dissect why it tries to go glance->oslo.messaging->d2to1 according to the log18:19
clarkbI think we will get better runtime numbers too since we don't cancel jobs in the check queue often18:19
ndipanovok well thanks guys - that clears some of things up18:19
*** dolphm has quit IRC18:19
jeblairclarkb: aprvd18:19
*** dolphm has joined #openstack-infra18:21
fungi also bailed on a java exception collecting mysql logs from the slave workspace. haven't seen that one before18:22
jeblairfungi: timeout18:22
jeblairfungi: was interrupted while uploading logs (so just barely timed out)18:23
ndipanovzul, any chance we'll see it in stackforge/gerrit soon?18:23
clarkbso uh the wheel stuff has broken the gate >_>18:23
fungijeblair: right, and devstack-precise-hpcloud-region-b-78802018:23
clarkb lots of that sort of thing18:23
fungiwhich explains the timeout18:23
zulmordred:  ^^^18:24
mordredhow did that break the gate?18:24
fungiclarkb: well, the gate is also completely broken today because django 1.6 doesn't work with... something tempest-wise18:24
*** dstanek has quit IRC18:24
*** cederb02 has joined #openstack-infra18:25
openstackgerritA change was merged to openstack-infra/config: Don't use new HPCloud regions in the gate.
markmcmordred, oslo.messaging 1.2.0 was never released, right18:25
markmcmordred, what's up?18:25
fungiclarkb: currently trying to figure out why the revert gor the change bringing in new django won't pass the requirements-integration test18:25
clarkbmordred: "Add support for per-distro wheel mirrors" did it I think18:25
reedwe need a gate before the gate18:25
mordredclarkb: right - but - that should be adding those as extra-index-urls - why is it not finding the things in the standard one18:26
*** dolphm has quit IRC18:26
clarkbmordred: oh gotcha, it should also look in the standard place18:26
clarkbpip bug?18:26
mordredclarkb: look before it - for pycrypto - we don't find it in wheel location - but we _do_ find it normally18:26
*** cederb02 has quit IRC18:26
reedwith the gate borked I feel bad about asking for a fix to the planet cache ... right?18:27
jeblairmordred: it's throwing lots of _ecxeptions_!18:27
mordredclarkb: ok. let's revert it18:27
*** oubiwann_ has joined #openstack-infra18:27
mordredsorry guys - I blame dstufft18:27
*** cederb02 has joined #openstack-infra18:27
reedhi oubiwann18:27
clarkboh lol the centos upgrade is causing the problems18:27
clarkbmordred: centos is now 6.518:27
mordredthat shouldn't matter18:27
oubiwann_reed: hello!18:27
clarkbmordred: I wonder if we just rerun the mirror if it will fix it18:27
mordredit should not matter18:27
jeblairyeah, it needs to not break in this situation18:27
mordrednope. there are two things broken there18:27
clarkbmordred: oh wait18:28
mordrednot having a wheel dir should not break anuthing18:28
mordredit's configured to look in both places18:28
mordredwhat it seems is happening18:28
clarkbmordred: is the dir not CentOS-6.5 and I agree it shouldn't completely break18:28
mordredis that something is causing case insensitivity searching to stop working18:28
mordredWHY IN THE WORLD that change is causing that behvior is gobsmackingly weird18:29
clarkbhmm sphinx vs Sphinx?18:29
mordredyah. that normally works18:29
mordredfor some reasons, it's not working here18:29
mordredeven though we did not change codepaths that shoudl affect that18:29
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Revert "Add support for per-distro wheel mirrors"
*** ^d has quit IRC18:29
*** ^d has joined #openstack-infra18:30
openstackgerritKhai Do proposed a change to openstack-infra/config: upgrade review-dev from gerrit ver 2.4.x to ver 2.8
mordredstab stab stab18:31
jeblairreed: what needs to happen?18:31
*** jamesmcarthur has quit IRC18:32
openstackgerritKhai Do proposed a change to openstack-infra/config: upgrade review-dev from gerrit ver 2.4.x to ver 2.8
portantemordred: our pep8 job failed with a UnicodeDecodeError, is that known?18:33
openstackgerritA change was merged to openstack-infra/config: Revert "Add support for per-distro wheel mirrors"
*** DennyZhang has joined #openstack-infra18:33
mordredportante: it's not known to me - point me at it?18:33
portante"No distributions at all found for openstack.nose-plugin"18:33
mordredoh - that's a whole other thing18:33
*** jerryz has joined #openstack-infra18:33
mordredthat's known - we're fixing it18:33
portantek, it seems to be the cause of a whole bunch of gate failures18:33
mordredor, at least, we've reverted the thing that we think caused it18:33
*** markmc has quit IRC18:34
portantek thx18:34
*** DennyZhang has quit IRC18:34
portanteis there a bug to recheck/reverify on?18:34
*** loq_mac has joined #openstack-infra18:34
*** jamesmcarthur has joined #openstack-infra18:34
reedjeblair, I misspelled Terri Yu's name, fixed the blog post but I fear the planet won't refresh unless it's manually triggered18:34
portantemordred: if so, just tell me yes, and I'll find it18:34
*** DennyZhang has joined #openstack-infra18:34
jeblair#status alert gate failures due to django incompatibility, pip bugs, and node performance problems18:34
openstackstatusNOTICE: gate failures due to django incompatibility, pip bugs, and node performance problems18:34
*** Abhishek_ has joined #openstack-infra18:34
*** ChanServ changes topic to "gate failures due to django incompatibility, pip bugs, and node performance problems"18:34
*** fbo_away is now known as fbo18:35
*** Abhishek_ has quit IRC18:36
*** loquacities has joined #openstack-infra18:36
jeblairreed: where is it incorrect? i see "As one of the recipient of the travel grant, Terri Yu told us:" which lgtm18:36
reedjeblair, oh, that's weird18:37
jeblairreed: (except "recipient" should be plural)18:37
*** harlowja has joined #openstack-infra18:37
reednever mind then18:37
jeblairreed: maybe it updated?  shift-reload?18:37
reedmaybe, yeah18:37
reedok, issue solved18:37
*** Abhishek_ has joined #openstack-infra18:37
*** loq_mac has quit IRC18:38
fungimordred: so, the line quoted in for that traceback was removed in change which was merged between 1.2.0a11 and 1.3.0a118:39
*** cederb02 has quit IRC18:41
fungimordred: and the requirement is for oslo.messaging>=1.2.0a11 so presumably it's building the minimum version matched there rather than the branch tip or latest tag18:41
mordredwhy would it build minimum ... wait18:41
clarkbcould that possibly be related to the pip weirdness with wheels?18:41
*** mancdaz is now known as mancdaz_away18:41
mordredGAH GAH GAH18:42
* mordred stab stab stab stab stab18:42
fungimordred: at least i'm otherwise at a loss for how the line, as quoted from, could in any way be present18:42
mordred>= is a misnomer18:42
fungioh, that18:42
mordredso that's why it's getting oslo.messaging old with d2to118:42
mordredNOW - why is that breaking then and not at other times18:42
clarkbbecause d2to1 is not in your local mirror18:43
fungibecause we're urgently trying to un-break things and the universe hates us for trying to reduce its entropy quotient18:43
clarkb:) is it possible some other dep used to pull in d2to1 and now doesn't?18:43
*** Ryan_Lane has quit IRC18:43
fungibut clarkb's probably on the nose there18:43
clarkbthis is why that change zaro wrote is handy, too bad we didn't have it in over the last few days18:43
*** ewindisch_ has joined #openstack-infra18:44
*** nati_ueno has quit IRC18:45
*** fbo is now known as fbo_away18:45
*** loquacities has quit IRC18:46
*** osanchez has quit IRC18:46
*** jerryz_ has joined #openstack-infra18:46
portantemordred: would that pip problem be fixed now? or does it take a while for that change to make it way through the system?18:49
jog0fungi: didn't merge18:49
fungijog0: yes, we're just about done figuring out why18:49
jog0clarkb: I have a seprate change that will help the region thing
*** jerryz has quit IRC18:50
mgagne60000! :O18:50
clarkbportante: it can take some time for it to get applied to all of the slaves. Change would've landed on puppet master ~5 minutes ago and puppet runs every 15 minutes so a little after the top of the hour we should be good18:50
jog0mgagne: \o/18:50
clarkbjog0: I don't think that change is necessary, we will stop running the large ops tests no those slaves instead18:50
*** dolphm has joined #openstack-infra18:50
portanteokay, thanks18:50
clarkbjog0: but your change doesn't hurt either so I won't -1 it18:50
jog0clarkb: I agree that  both patches aren't necessary but I think its safer to lower that number18:51
*** oubiwann_ has quit IRC18:51
jog0having at 150 isn't helping anything as there are no efforts to get nova and neutron to work better (at least not priority ones)18:51
jog0and 100 is still a good regression test (we couldn't do 50 before)18:51
fungimordred: so should the requirements-integration job, in the new landscape once wheels are being built, start ignoring -f lines while the projects work on removing them?18:51
*** dolphm has quit IRC18:52
jog0oslo.messaging fun18:52
*** dolphm has joined #openstack-infra18:52
mordredfungi: yeah - but I don't think that's our problem right now18:52
jerryz_clarkb jog0: since you are here, i want to check if elastic-recheck has an exception exit issue. My recheck bot sometimes will end with this log:
*** mrodden has quit IRC18:53
fungijog0: yes, glance wants an older oslo.messaging version at the moment, which in turn wants d2to1 which we're not locally mirroring in the process of requirements-integration tests18:53
clarkbfungi: just add d2to1 to global reqs in that change?18:53
jog0fungi: one step forward one step back18:53
clarkbthen we can fix glance18:54
clarkbthis skirts around the problem. but should move us forward18:54
mordredthe real questoin is - what caused the local mirror to suddently stop containing d2to118:54
mordredI mean, give it a try18:54
*** oubiwann_ has joined #openstack-infra18:54
jog0jerryz_: looks like an irc connection issue at first glance18:54
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Nodepool: cache all gerrit repos
*** krotscheck has joined #openstack-infra18:55
jerryz_jog0: do we have a way to maintain the connection?18:55
jerryz_or how the channel ends up being closed?18:55
jog0jerryz_:  I am not really sure actually I didn't work on the bot mtreinish did18:56
*** ryanpetrello has quit IRC18:56
*** michchap has joined #openstack-infra18:56
clarkbjerryz_: I disconnected from freenode last night could that have been what you saw?18:56
*** dolphm has quit IRC18:57
clarkbalso it depends on so many things, which server you are connecting to, local firewalls and so on18:57
*** ryanpetrello has joined #openstack-infra18:58
jerryz_i just named a channel and let recheck join.18:58
jerryz_there are only the bot and me in the channel18:58
*** oubiwann_ has quit IRC18:58
*** mrodden has joined #openstack-infra18:59
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6"
fungiclarkb: mordred: ^ trying the suggestion18:59
jog0fungi: haha i like the hack19:00
jeblairfungi: er, isn't d2to1 in the mirror?19:00
fungijeblair: but not in the "local" mirror the integration test builds on-demand for that job19:00
mordredjeblair: it's not showing up in the locally built mirror in in the requirements job19:00
mordredwhich is bonghits19:00
mordredbut we don't know why yet, and we don't know why any changes to that process that caused it to not be there wound up passing tests19:01
jeblairmordred: okay, so how has _that_ not tripped us up before?19:01
*** michchap has quit IRC19:01
mordredthat's my question19:01
openstackgerritlifeless proposed a change to openstack-infra/reviewstats: Remove arata and devananda from TripleO core.
mordredunless we were not skipping mirror properly because of the other change we looked at19:01
*** pete5 has quit IRC19:02
jog0lifeless: why not hook reviewstats into gerrit groups to know core?19:02
*** pete5 has joined #openstack-infra19:02
*** pete5 has quit IRC19:02
fungijog0: gerrit 2.4.x provides no mechanism to query group membership19:02
*** pete5 has joined #openstack-infra19:02
lifelessjog0: patches appreciated etc etc etc19:02
*** wenlock has quit IRC19:02
jog0fungi: lol besides screen scraping?19:02
fungiwell, yeah... that *shudder*19:03
lifelessmikal: you get group infro from gerrit, don't you?19:03
*** talluri has joined #openstack-infra19:04
lifelessanyone have ideas about the py33 failure on checks ?19:04
*** syerrapragada1 has quit IRC19:04
*** syerrapragada has joined #openstack-infra19:04
fungijog0: also, it's not as simple as "members of some group" (you need to collect the approval acl entries from the project(s) of interest, then descend recursively through the referenced groups since gerrit allows arbitrary group inclusion in other groups)19:04
jog0fungi: ohh gerrit19:06
openstackgerritA change was merged to openstack-infra/jeepyb: Always sleep after group creation
fungilifeless: Non-zero exit code (2) from test listing. stdout=b'\xb3 `@R\x17text/plain;charset=utf8\rimport errors"reviewstats.tests.test_projectinfo\xc8\x19\xce\xfe', stderr=None19:08
openstackgerritClark Boylan proposed a change to openstack-infra/config: Remove hpcloud region-a from nodepool.
clarkbmordred: ^19:08
fungilifeless: that's a new one on me19:08
mordredclarkb: thanks19:08
jog0clarkb: just out of curiosity why remove new region all together?19:09
jerryz_clarkb: i see irc has an ping-pong heartbeat mechanism. if the server doesn't get a pong for a ping request, the connection to the client will be closed. i suspect somehow either ping or pong didn't get through firewall. but e-r is not supposed to die in this scenario19:10
fungiclarkb: mordred: do we think region a is unsuitable but region b is fine?19:10
clarkbmordred: ^19:10
*** ^d has quit IRC19:10
lifelessfungi: oh right, my eyes glazes on that19:10
clarkbjerryz_: right, it should probably try to reconnect with an expoential backoff19:10
mordredfungi: request from HP - region-a is under pressure and we should really just stick with region-b for the new cloud19:10
lifelessfungi: I have a patch to make that nicer, but it at least lets me know what to poke at19:11
mordredfungi: I probably should have known that already before we added a in - so it's my bad19:11
fungimordred: worth noting, we're getting similarly slow performance on region b. presumably this is not intended to address that19:11
lifelessfungi: it means discover failed on py3319:11
mordredit is not19:11
lifelessfungi: but where did stderr for it go?19:11
jog0mordred: ahh19:11
fungilifeless: that is indeed a very, very good question19:12
lifelessfungi: testr in that version calls subprocess with stderr undefined, so it will be on the processes fd 219:12
jeblairclarkb: fyi your change will cause nodepool to fail to clean up anything in region-a19:12
jeblairclarkb: so you may want to 'nova delete' things manually19:12
lifelessfungi:'s hook does19:12
lifeless        return commands.run_argv([sys.argv[0]] + list(args),19:13
lifeless                                 sys.stdin, sys.stdout, sys.stderr)19:13
lifelessso it should also be wired directly to fd219:13
clarkbjeblair: ok, will the nodepool DB need cleaning up too?19:13
clarkbjeblair: should I just remove region-a from the jenkins masters and leave the creds there?19:13
jeblairclarkb: that would be the cleanest way19:13
clarkbI just -2'd the change, will update19:14
fungiclarkb: jeblair: could we reduce the quota to 0 with a patch first, and then remove once settled?19:14
*** wenlock has joined #openstack-infra19:14
*** johnthetubaguy has quit IRC19:14
ArxCruzjeblair: when you got time, check if is what you want :)19:14
jeblairfungi: i think removing it from the masters will be sufficient.  _also_ setting quota to 0 shouldn't hurt.19:15
*** ^d has joined #openstack-infra19:15
*** ^d has quit IRC19:15
*** ^d has joined #openstack-infra19:15
openstackgerritClark Boylan proposed a change to openstack-infra/config: Stop using hpcloud region-a.
jeblairArxCruz: will do; there's a lot going on right now.19:16
ArxCruzi know, when you got chance ;)19:17
*** thomasem has quit IRC19:17
*** senk has joined #openstack-infra19:18
*** markwash has quit IRC19:19
*** senk has quit IRC19:20
*** senk has joined #openstack-infra19:20
*** fbo_away is now known as fbo19:21
openstackgerritJohn Dickinson proposed a change to openstack-infra/reviewstats: update swift core
*** talluri has quit IRC19:23
clarkbI need to AFK for a bit during the middle of the day. Going to visit the dentist. Will do my best to keep up with the fires19:24
*** SergeyLukjanov has joined #openstack-infra19:24
*** SergeyLukjanov has quit IRC19:24
*** SergeyLukjanov has joined #openstack-infra19:25
fungiyep, and i'm disappearing for a few hours starting a little after 2100 utc (monthly local osug meetup)19:25
clarkbfungi: I see tempest tests are running and passing. is django 1.6 only a problem occasionally?19:25
fungiclarkb: on full, postgres-full and neutron i think19:26
jeblairfungi, mordred: can you summarize where you are on the django thing?19:26
sdagueclarkb: where do you see passes?19:26
*** vkozhukalov has quit IRC19:26
clarkboh I see now19:26
mordredjeblair: fungi uploaded a modified change and we're watching to see if it passes build19:26
clarkbsdague: I was looking at largeops and things not realizing an actual test was failing19:26
clarkbI thought it was pure dependency weirdness19:27
*** nicedice has joined #openstack-infra19:27
sdagueok, do we have a bug on this yet?19:27
*** sarob has joined #openstack-infra19:27
sdagueI was going to write the er for it19:27
clarkbI didn't write one19:27
fungithough jenkins/zuul is estimating 51 minutes remaining on
clarkbmordred: want to approve ?19:28
mordredclarkb: done. thanks19:29
jeblairi approved 60014 because it's silly to wait on it in check19:30
fungifair enough19:31
lifelessrussellb: python3 we'll need to make non-voting I think19:31
uvirtbotLaunchpad bug 1257885 in horizon "horizon does not work with django 1.6 " [Undecided,New]19:31
lifelessrussellb: paramiko :(.19:31
russellblifeless: ah ok, fine with me19:31
russellblifeless: thanks :)19:31
openstackgerritA change was merged to openstack-infra/config: Stop using hpcloud region-a.
jeblairlifeless: why not remove it?19:32
sdaguejeblair: agreed, honestly, you should kill everything else in the zuul queue19:32
sdaguebecause they are all going to fail until that goes in19:32
lifelessjeblair: I've fixed all the other bits I could find, when paramiko merges the 3 series patch it should just work19:32
fungisdague: we don't know for sure that it's going to succeed either19:32
fungisdague: but yeah19:32
lifelessjeblair: and we have this openstack wide desire to support 3 well, don't we?19:32
sdaguefungi: sure, but we know everything else is going to fail19:32
jeblairlifeless: oh, ok, so there is hope.  nonvoting is fine then.19:32
sdaguemight as well get rid of it19:32
*** ^demon has joined #openstack-infra19:33
*** rpodolyaka1 has joined #openstack-infra19:33
jeblairsdague, fungi: i will make it happen19:33
mgagnejeblair: is /home/gerrit2/acls/inactive.config a built-in ACL? won't gerrit crash if acl config does not exist?19:35
mgagnejeblair: or is it handled by jeepyb?19:35
openstackgerritlifeless proposed a change to openstack-infra/config: Make python3.3 nonvoting for reviewstats.
jeblairmgagne: doesn't exist yet.  would you mind writing it as part of that change?19:35
fungii suppose i could remove django 1.6 from as a temporary measure to reduce pain19:35
mgagnejeblair: how are projects deprecated in openstack?19:35
*** ^d has quit IRC19:36
sdagueclarkb: we don't index horizon logs yet eh? is that because of the os-loganalyze filter?19:36
jeblairmgagne: we haven't really done it yet.  we should probably do something similar for melange.19:37
jeblairmgagne: that's the only one i think.19:37
*** dcramer_ has joined #openstack-infra19:37
*** ^demon has quit IRC19:37
*** mrodden has quit IRC19:39
*** fbo is now known as fbo_away19:39
fungigiven that previous tests of 60014 passed tempest tests which were failing without it, i think it's safe to say it will solve the django situation once merged... so any reason not to delete static.o.o:/srv/static/pypi/openstack/Django/Django-1.6.tar.gz in the meantime?19:40
jeblairfungi: wfm19:40
* fungi does19:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add query for the horizon gate wedge
jeblair60014,2 is a head now19:41
sdaguejeblair: so do you think this is close enough to resolving that we don't need to land that er check?19:42
jeblairsdague: i'd do it; there's still a bunch of check jobs outstanding, and it may catch some of the gate jobs still running19:42
*** mrodden has joined #openstack-infra19:42
sdaguejeblair: ok, landing19:43
openstackgerritIvan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge
openstackgerritA change was merged to openstack-infra/elastic-recheck: add query for the horizon gate wedge
*** wenlock has quit IRC19:46
jerryz_fungi: did you ever turn up the flavor for devstack slaves?  Recently I have been noticing in my own CI environment ,tempest is prone to failures possibly caused by  race condition bugs. after i increase the number of cores for devstack slaves, most of the failures are gone.19:47
*** bknudson has joined #openstack-infra19:47
jerryz_fungi: is tempest in parallel demanding more performance than ever?19:47
fungijerryz_: i don't recall being asked, but we run 8gb flavor slaces19:48
fungier, slaves19:48
fungiso that's 4 vcpus and 8gb ram19:49
jerryz_jerryz: my ram is 8gb too. i just increased the number of cpus19:49
jerryz_jerryz: so i wonder what is the type of physical cpu hp cloud uses?19:49
*** Mithrandir has quit IRC19:49
fungijerryz_: sounds likely to affect it, sure. in this case i think we consider the sorts of race conditions we expose in openstack by putting resource pressure on the slave vms to represent actual errors in openstack we want to fix19:50
jerryz_fungi: mine is 2.4Ghz intel 562019:51
openstackgerritSean Dague proposed a change to openstack-infra/config: create a selection of jobs to not use config drive
*** dolphm has joined #openstack-infra19:53
*** Mithrandir has joined #openstack-infra19:53
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: allow configdrive vs. metadata server to be configured
sdaguejeblair: so if 60061 lands first, then 59303 is actually self testing19:55
*** dcramer_ has quit IRC19:56
*** miqui has joined #openstack-infra19:56
*** michchap has joined #openstack-infra19:57
*** dolphm has quit IRC19:57
miquizuul users.....seen this before
mordredfungi: failed19:58
miquifungi or mordred...perhaps you have seen this one
jeblairmiqui: perhaps it's a draft change19:59
fungimordred: gah, so is that local mirror being built by installing the global requirements list, or by installing the integrated projects?20:00
fungimordred: if the latter, then if no project is directly installing d2to1 perhaps it doesn't end up in the local mirror copy?20:00
miquihmm ..jeblair... what is interesting is the KeyError: 'project'20:01
mordredfungi: it's built by running pypi-mirror against the requirements repo20:01
clarkbI don't see the failure20:01
jeblairclarkb: check failure20:01
*** senk has quit IRC20:02
*** senk has joined #openstack-infra20:02
anteayaclarkb: okay, fungi has left some comments so I'll talk to EmilienM about addressing those and then see where we are about a plan to move forward20:02
*** michchap has quit IRC20:02
clarkbanother work around would be to remove newer django from the mirror (to get other changes to work)20:02
jeblairclarkb: fungi has done that20:03
fungiclarkb: i did20:03
anteayagrenade for Neutron is one of our must haves so we have to move forward on it as part of our testing mandate20:03
fungiclarkb: hopefully jobs starting after 1940 utc will get django 1.5.520:03
fungiclarkb: at least until the next mirror run, if we don't get this revert through20:03
jeblairsince everyone is about to leave, how about we just merge the revert.20:04
*** boris-42 has quit IRC20:04
fungijeblair: i'll upload a switch back to ps1 for it then20:04
openstackgerritJeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6"
fungithat's ^ back to just the plain revert, no d2to1 monkeybusiness20:05
openstackgerritA change was merged to openstack/requirements: Revert "Allow Django 1.5 and 1.6"
*** sarob has quit IRC20:06
*** herndon_ has joined #openstack-infra20:06
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Revert "Use wheels for installation"
*** sarob has joined #openstack-infra20:06
*** herndon_ has left #openstack-infra20:06
mordredthat ^^ is the only other change that's happened that I could see causing this20:06
*** iv_m has joined #openstack-infra20:06
mordredof course, why it itself wouldn't have failed is behond me20:06
*** oubiwann_ has joined #openstack-infra20:07
*** dolphm has joined #openstack-infra20:08
ArxCruzclarkb: fungi mordred is the check-tempest-dsvm-neutron always failing ?20:08
fungiArxCruz: if it's the 500 internal server error pattern, recheck bug 125780920:09
uvirtbotLaunchpad bug 1257809 in openstack-ci "Django 1.6 found incompatible during integration testing" [Critical,Fix released]
*** derekh has joined #openstack-infra20:10
*** dstanek has joined #openstack-infra20:10
*** sarob has quit IRC20:10
fungithough i think elastic-recheck may be starting to report it as bug 1257885 instead20:10
uvirtbotLaunchpad bug 1257885 in horizon "horizon does not work with django 1.6 " [High,Triaged]
mordredfungi, jeblair: so - summary of things we've learned is, I guess, before we had a bug in requirements testing which caused 1.6 django to get through untested. somehwere in the midst of addressing that, something broke all requirements integration testing20:11
openstackgerritStuart Warren proposed a change to openstack-infra/jenkins-job-builder: Add Mercurial plugin feature to jenkins-job-builder
mordredso if we revert enough things, we should be back to just "reuiqrements testing is broken as before"20:11
jeblairfungi: i marked the bug as a dup.20:11
mordreddoes that sound right to you guys?20:11
mordredI'm thinking a next step, if it starts working again20:12
zarois it possible to clone a non-master branch with gerrit-git-prep?20:12
dkranzIs the persistent "ImageKilledException: Image dc0db32f-5da6-4125-a143-39948c8b22be 'killed' while waiting for 'active'" on the radar?20:12
*** sarob has joined #openstack-infra20:12
*** Abhishek_ has quit IRC20:12
fungimordred: yeah. i mean, if you revert enough things you'll be back to commit 0x020:12
jeblairmordred: yes.  i'd like to get the mirror selection changes in.20:12
mordredmaybe to merge just the select mirror logic _without_ the enable wheel change - thinking that perhaps the enable wheel change is what's making things bork20:12
dkranzI had not seen that before but it just started happening a huge amount20:12
mordredeven though its ability to make things bork seems very strange20:12
fungidkranz: have an example log?20:14
jeblairnnfi doesn't seem to play well with a merged change in the queue :(20:14
jog0dkranz: in general lets try to keep the is bug x known in the qa room being its qa related20:14
jog0dkranz: but while here20:14
jog0dkranz: I was waiting for gate to unwedge before digging into that20:15
dkranzActually perhaps it is the same as the http 500 failure which also shows in the log20:15
jog0our baseline is very dirty now so figured better to wait a bit20:15
*** dstanek has quit IRC20:15
jeblairmordred, fungi, clarkb: please review so that it can get picked up in the zuul restart i'm about to do.20:15
fungidkranz: so probably django 1.620:15
mordredjeblair: looking20:16
dkranzfungi: Perhaps. BUt the stacktrace is from a different test. Ugh20:16
dkranzfungi: But they started happening about the same time so probably related.20:17
*** sarob has quit IRC20:17
*** dstanek has joined #openstack-infra20:17
dkranzjog0: ok20:17
jog0dkranz: yeah I am not convinced its related but too much noise right now so figured better wait20:17
jog0I thought that a few hours ago though20:17
jog0but we should be out of the water soon20:18
sdaguethe code size bar on new gerrit is super nice -,n,z can't wait until we get new gerrit in infra20:18
*** iv_m has left #openstack-infra20:18
mordredjeblair: fyi - your lambda function can be replaced with key=operator.itemgetter('name')20:19
mordredjeblair: but I don't care20:19
jog0sdague: that is really slick20:19
mordredjeblair: more of a "I learned about that function recently, so I thought I'd share" - +A20:19
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: remove space that fails to have this check match
jog0so merged now we expect things to pass?20:20
fungijog0: since deleting django 1.6 from the mirror at 1940 utc we expect jobs starting after that time to pass20:21
*** boris-42 has joined #openstack-infra20:22
openstackgerritA change was merged to openstack-infra/elastic-recheck: remove space that fails to have this check match
*** dstanek has quit IRC20:23
jog0fungi: cool I can see dropping already20:23
jeblairmordred: lifeless pointed that out as well.  it would require an import.  also, it's attrgetter.  tbh, i find the lambda easier in general, but will consider attrgetter in the future if i need performance or sorting on multiple attributes.20:24
*** Abhishek_ has joined #openstack-infra20:24
*** che-arne has joined #openstack-infra20:24
mordredjeblair: nod20:24
jamesmcarthurHi - is sdague around today?20:26
sdaguejamesmcarthur: yes20:26
sdaguesorry, it was late last night when you were looking for me20:26
jamesmcarthursdague: no worries! i'm troubleshooting the problem you reported with the member profile/affiliations on openstack.org20:27
jamesmcarthurI've had a lot of trouble replicating the errors you were seeing.  However, after some analysis, we identified a couple of problems, which I believe we resolved early this morning.20:28
*** melwitt has joined #openstack-infra20:28
jamesmcarthurI'm not 100% they're related to the ajax timeouts you were seeing, but I was hoping to check with you again.20:28
*** loq_mac has joined #openstack-infra20:29
*** dstanek has joined #openstack-infra20:29
jamesmcarthurIf your'e still running into issues, I'd like to set up a time to troubleshoot with you, at your convenience.20:29
fungizaro: if nobody answered your question yet, gerrit-git-prep resets to whatever branch the change is being applied to, so if you submit a change to the openstack/2.8 branch of openstack-infra/gerrit for example, that's what it should check out to run jobs for it20:31
sdaguejamesmcarthur: ok, let me give it a shot20:31
jamesmcarthursdague: thanks!20:31
*** loquacities has joined #openstack-infra20:31
fungizaro: more accurately, it should check out whatever ref zuul has constructed and tells it to use20:33
*** ^d has joined #openstack-infra20:33
fungibut zuul takes the gerrit branch for the change into account to determine how to build the ref20:33
*** loq_mac has quit IRC20:33
openstackgerritA change was merged to openstack-infra/zuul: Fix merging transitively-connected change queues.
miqui@jeblair  ---> <jeblair> miqui: perhaps it's a draft change20:35
miquican you elaborate?20:35
jeblairmiqui: it probably got bad data back from gerrit; that might happen if it's a draft change20:35
*** sarob has joined #openstack-infra20:37
openstackgerritA change was merged to openstack-infra/zuul: Add a zuul client
devanandahi guys! how would i get ironic added to ?20:37
*** sarob_ has joined #openstack-infra20:38
*** sarob has quit IRC20:39
*** sarob_ has quit IRC20:40
*** Abhishek_ has quit IRC20:40
*** sarob has joined #openstack-infra20:40
miqui..thanks jeblair...20:40
*** senk has quit IRC20:41
russellblifeless: did you already submit the change to make py33 non-voting for reviewstats?  if not i can20:41
jeblairrestarting zuul20:42
*** _david_ has joined #openstack-infra20:42
*** loquacities has quit IRC20:42
*** jerryz_ has quit IRC20:43
mgagnedevananda: my understanding is that this page is generated by Project list is hardcoded here:
devanandamgagne: perfect. thanks20:44
SpamapS2013-12-04 18:47:54.077 | No distributions at all found for sphinx>=1.1.2 (from -r /home/jenkins/workspace/gate-os-apply-config-python26/test-requirements.txt (line 7))  ??20:44
jeblairzuul is running, and i have re-enqueued the check and gate jobs using the new script (!)20:44
*** sarob has quit IRC20:45
fungiyummy, yummy dogfood!20:45
*** sarob_ has joined #openstack-infra20:45
portantejeblair: is there a bug number that I can recheck jobs against?20:45
openstackgerritDevananda van der Veen proposed a change to openstack-infra/reviewday: Add Ironic projects
portantefor the above "no distributions ..."20:45
fungiportante: for which specific failures?20:45
mordredSpamapS: yeah - that's evil we're working on fixing right now20:46
mordredfungi: I suppose maybe I should file a bug?20:46
fungisounds like there isn't one yet20:46
portantealmost all of our most recent jobs have been hosed by the no distributions20:46
jeblairmordred: is that the pbr revert?20:46
*** derekh has quit IRC20:46
mordredjeblair: yeah. at least I hope it is20:47
jeblairmordred: i put it at the top of the queue20:47
*** sarob has joined #openstack-infra20:47
*** sarob_ has quit IRC20:48
mordredSpamapS, portante: bug 125790520:48
uvirtbotLaunchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New]
portantemordred: thanks, should I recheck or wait?20:49
fungiportante: i think we're still waiting for openstack-dev/pbr 60063,1 to merge20:49
portanteokay, i'll hold off then20:49
fungiit's got 51 minutes by the status page estimate20:49
fungiand no changes ahead of it20:50
*** dprince has quit IRC20:50
portanteonce it merges, will it take long to get deployed?20:51
*** vipul is now known as vipul-away20:51
SpamapSmordred: ty20:51
fungiportante: i think once that change merges any jobs started after that should get the old (working) behavior again20:51
mordredfungi: if it's still broken at that point, I'm going to rewrite everytihng in go20:52
clarkband rust20:53
fungimordred: static linking will solve all our problems20:53
*** MarkAtwood has joined #openstack-infra20:55
sdaguemordred: man, you should just do asm.js to get full baremetal performance20:55
mordredI don't even want to know what you just said20:57
jeblairmordred: help me out here... on
mordredjeblair: looking20:58
jeblairmordred: why do i keep seeing 'pip wheel' commands?20:58
*** hashar has joined #openstack-infra20:58
mordredjeblair: that builds the wheels20:58
mordredwhich is a function in pypi-mirror itself20:58
*** markwash has joined #openstack-infra20:58
jeblairah i see20:59
mordredjeblair: btw - I think there may be a bug somewhere in our mirror building20:59
mordredjeblair: like, the production mirror building20:59
mordredI need to go look at logs20:59
mordredbut the fact that we do not have a centos 6.5 dir published on the mirror is concerning20:59
openstackgerritA change was merged to openstack-infra/reviewday: Add Ironic projects
clarkbmordred why? didnt you add that a while back?21:01
mordredclarkb: yup. that's what it's concerning that it's not there21:02
clarkboh that is because centos 6.5 is brand new21:02
clarkbreleased yesterday. probably need to rerun mirror job21:02
*** wenlock has joined #openstack-infra21:05
*** SergeyLukjanov has quit IRC21:05
*** esker has quit IRC21:06
openstackgerritA change was merged to openstack-infra/config: Remove braces from bash vars in solum job config
*** sandy__ has joined #openstack-infra21:08
*** sandywalsh_ has quit IRC21:08
*** harlowja has quit IRC21:09
*** katyafervent has quit IRC21:09
*** katyafervent has joined #openstack-infra21:09
*** harlowja has joined #openstack-infra21:09
*** DennyZhang has quit IRC21:10
*** smarcet has left #openstack-infra21:11
fungiokay, heading out, but will be semi-on from the cisco offices in a bit21:15
mordredclarkb: ok. I'll hold off on panicing on that so far21:15
*** denis_makogon has quit IRC21:17
*** paul-- has joined #openstack-infra21:17
*** loq_mac has joined #openstack-infra21:17
*** dcramer_ has joined #openstack-infra21:18
*** loquacities has joined #openstack-infra21:20
zarofungi: thnx for answering my question about gerrit-git-prep.21:21
jeblairmordred, fungi: so at least one gate job has fully succeeded (#3)21:21
jeblair(behind the pbr job)21:21
mordredjeblair: thank god21:21
*** loq_mac has quit IRC21:22
*** Mithrandir has quit IRC21:24
*** Mithrandir has joined #openstack-infra21:24
*** julim has quit IRC21:25
*** loquacities has quit IRC21:25
*** vipul-away is now known as vipul21:25
*** Abhishek_ has joined #openstack-infra21:27
*** dstanek has quit IRC21:28
*** dstanek has joined #openstack-infra21:29
*** dstanek_afk has quit IRC21:30
*** dkranz has quit IRC21:30
*** dstanek has quit IRC21:30
openstackgerritDavid Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support
*** dstanek has joined #openstack-infra21:32
*** dstanek has quit IRC21:33
*** thomasem has joined #openstack-infra21:33
*** thomasem has quit IRC21:34
*** thomasem has joined #openstack-infra21:34
*** hashar has quit IRC21:35
*** dstanek has joined #openstack-infra21:36
*** loq_mac has joined #openstack-infra21:37
*** paul-- has quit IRC21:37
*** sarob has quit IRC21:39
*** sarob has joined #openstack-infra21:39
openstackgerritSteve Baker proposed a change to openstack-infra/config: Enable neutron on heat-slow experimental job
*** dkliban_ has quit IRC21:39
jeblairrestarted nodepool to get that logging change in21:42
*** sarob has quit IRC21:43
openstackgerritSean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template
*** dolphm has quit IRC21:45
openstackgerritSean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template
*** rongze has quit IRC21:49
*** Abhishek_ has quit IRC21:51
*** Abhishek_ has joined #openstack-infra21:51
*** CaptTofu has quit IRC21:53
*** Abhishe__ has joined #openstack-infra21:54
lifelessrussellb: I did21:54
*** Abhishek_ has quit IRC21:55
fungizuul status is indeed looking far greener21:55
*** loq_mac has quit IRC21:55
*** loq_mac has joined #openstack-infra21:55
*** jamesmcarthur has quit IRC21:56
sdaguefungi: jog0's fun graph is definitely a winner here -
jeblairmordred, fungi: good news and bad news.  bad: the pbr change still failed gate-requirements-integration-dsvm good: it doesn't seem to be required for most changes21:59
jeblairmordred, fungi: so i think the current status is that openstack/requirements is hosed, but nothing else is.22:00
portantejeblair: is that one "timed out" doing pip install from
*** loq_mac has quit IRC22:01
*** loq_mac has joined #openstack-infra22:01
portanteis there a bug number for that that I can recheck against?22:01
mordredjeblair: and we force-landed the django change, yeah?22:02
jeblairmordred: yes22:02
mordredjeblair: so, for the moment, the fact that the requirements job is hosed is bad but not project-wide-critical22:02
*** harlowja has quit IRC22:02
portantethe pbr failure for swift hit this one:
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: allow configdrive vs. metadata server to be configured
jeblairmordred: yes, though i don't expect that to be the case for very long.22:03
*** mfer has quit IRC22:03
mordredme either22:03
jeblairportante: i don't know the answer to your question.  perhaps mordred does.22:04
jeblair#status ok22:05
*** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs | Bugs | Code"22:05
ekarlsoany devstackers here ?
portantejeblair, mordred, so then I'll file another bug the above pbr failure22:05
ekarlsois my local.conf, but it produces a " = " line in my nova.conf ?!22:05
uvirtbotLaunchpad bug 1257933 in devstack "pbr fails to install during check-swift-dsvm-functional job" [Undecided,New]22:08
*** mfer has joined #openstack-infra22:08
*** dims has quit IRC22:10
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Make havana devstack/tempest jobs non-voting
*** dcramer_ has quit IRC22:13
*** pabelanger has quit IRC22:15
devanandareviewday question -- are there separate per-project pages, or just the one giant shared page?22:15
pleia2devananda: giant shared page, but we've been thinking about doing some per-project motifications now that it's so huge22:17
pleia2you know, in our free time :)22:17
devanandapleia2: awesome. so, like, tomorrow, right? :)22:17
pleia2maybe by j-122:18
pleia2actually, I should create a bug for it so it's more formally on my radar22:18
*** rongze has joined #openstack-infra22:19
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add support for batch_tasks.
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add seealso to batch_tasks from promoted_build.
stevebakerI wonder if I could get some eyeballs on this?
openstackgerritDavid Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Activate new change screen per default
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Zuul: set gear log level to INFO
Alex_Gaynorsdague: If there's any particular django regerssion I could be hlpi9ng with, let me know22:21
_david_Gerrit 2.8: i wonder if we want to extend Gerrit TopMenu? As of Gerrit 2.8, it is possible to extend Gerrit Top Menu, We could put there Zuul of whatever we want22:22
*** loq_mac has quit IRC22:23
Alex_Gaynorcc: mordred, fungi ^ :-)22:23
_david_Example of gerrit Top-Menu extension in LibreOffice project: Buildbot coming from our buildbot plugin22:24
fungiAlex_Gaynor: i suspect there are now extensive logs archived from our accidentally running tempest with django 1.6 today22:24
Alex_GaynorAnyone have a link to one? :-)22:25
fungii'll dig one out22:25
*** dangers is now known as danger_fo_away22:25
*** harlowja has joined #openstack-infra22:27
fungiAlex_Gaynor: i guess for starters, the horizon crowd are tracking horizon support for django 1.6 at
jeblair_david_: the libreoffice logo owerwrites the search box if your screen is <1430 pixels22:27
fungiAlex_Gaynor: it sounds like the move to uncap it in requirements was probably premature22:27
*** rongze has quit IRC22:28
Alex_Gaynorseems so22:28
_david_jeblair, yeah ,-) a known issue22:28
_david_jeblair, back to top menu extension: do you think it would be helpful?22:28
*** loq_mac has joined #openstack-infra22:28
_david_jeblair, to have it, we would need a trivial gerrit plugin, but it would open a door for future extensions as well, ...22:29
jeblair_david_: i don't think it's a priority.22:30
clarkbI am mostly back now (eating a late lunch but in front of a computer)22:31
_david_jeblair, ok, just to let you know: you could put ACL logic on menu entries, so that only some groups can see some entries.22:31
clarkbwe are reverting the pbr integration with wheels change?22:31
jeblairclarkb: it failed requirements integration test.22:32
clarkbjeblair: is that what was causing the d2to1 madness?22:32
jeblairclarkb: no idea.  mordred: ?22:32
Alex_Gaynorfungi: do you know if there are issues besides the one review request that was merged?22:32
mordredclarkb: I do not know. none of the symptoms make any sense to me- I need to dive in much more deeply22:33
_david_jeblair, mordred what is the status with WIP plugin and needed core patch for 2.8?22:33
clarkbmordred: ok22:33
mordredzaro: ^^22:33
clarkbis there anything around the gate that I should be looking at or doing? or am I good to go back to reviews?22:33
*** rpodolyaka1 has quit IRC22:33
*** gyee has quit IRC22:33
_david_if we don't want/need it i'm fine with it22:34
clarkbin other news, last night the author of this change says they did not receive any email when jenkins and tom commented on that change, but do receive emails when they comment on changes by other authors. Any idea why that may happen?22:34
clarkb_david_: the needed core patch merged the other day so will be in 2.9. I think our hope is that 2.9 will release relatively quickly and we can use that22:35
fungiAlex_Gaynor: so, the issue was that we spontaneously started trying to use django 1.6 in tempest tests, if that's what you mean by "the one review request that was merged". the end result was a lot of 500 errors from the dashboard as seen in
clarkb_david_: otherwise we might use a build off of master or a minimal fork of 2.8 (but really want to use 2.9)22:35
Alex_Gaynorfungi: is there a way to see the horizon logs for the 500s in that log?22:35
clarkbAlex_Gaynor: yes
Alex_Gaynorclarkb: danke22:36
_david_clarkb, i guess we are talking about different patches? you mean zaro's change owner patch. It is entirely unrelated to my WIP plugin and needed core patch to make it work, that was rejected by mfick22:37
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/requirements: Add markdown to requirements
clarkbfungi: Alex_Gaynor looking at I don't understand why we couldn't get that stuff in before we do django 1.622:37
clarkb_david_: oh I thought your change for WIP is already in, I am talking about different things22:38
_david_clarkb, np, yeah the plugin is in (because /me have a +2 on gerrit-review for this plugin)22:38
*** ryanpetrello has quit IRC22:39
openstackgerritJaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add seealso to batch_tasks from promoted_build.
*** loquacities has joined #openstack-infra22:42
*** loq_mac has quit IRC22:42
*** fifieldt has joined #openstack-infra22:42
*** dkliban_ has joined #openstack-infra22:44
openstackgerritDouglas Mendizabal proposed a change to openstack-infra/config: Initial configuration for python-barbicanclient
*** miqui has quit IRC22:45
*** wenlock has quit IRC22:47
clarkbmordred: shoudl I trigger the python26 mirror job so that we can see if a centos 6.5 dir is created?22:48
*** Ryan_Lane has joined #openstack-infra22:49
*** rossella_s has joined #openstack-infra22:49
*** sarob has joined #openstack-infra22:50
SpamapS1386197303.447  15015 TCP_MISS_ABORTED/000 0 GET - HIER_NONE/- -22:51
*** loquacities has quit IRC22:51
SpamapS1386197318.458  15007 TCP_MISS_ABORTED/000 0 GET - HIER_NONE/- -22:51
SpamapSis just down?22:51
*** loq_mac has joined #openstack-infra22:51
SpamapS1386197323.436   4974 TCP_MISS/503 3723 GET - HIER_DIRECT/2001:4800:7813:516:3bc3:d7f6:ff05:4882 text/html22:51
clarkbSpamapS: can you link to the job that resulted in that22:53
clarkbor is this local to you?22:53
*** yjiang5_1 has quit IRC22:53
clarkbnevermind 192.168/16 would be local to you22:53
SpamapSclarkb: thats just me doing a 'tox -r -epep8' in openstack/tripleo-image-elements through my proxy with PIP_INDEX_URL=
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Fix CSV output
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Remove text blobs from CSV output
clarkbseems up for me22:53
SpamapSclarkb: yeah it isn't doing it now22:54
*** sarob has quit IRC22:54
* fungi blames internets22:54
clarkb is empty22:54
*** rongze has joined #openstack-infra22:55
lifelessclarkb: can I get an APRV on ?22:55
*** dkranz has joined #openstack-infra22:55
clarkblifeless: sure; hwoever, will we have to revert in an hour when paramiko is happy?22:56
lifelessclarkb: no22:56
lifelessclarkb: weeks probably22:56
lifelessclarkb: the patch set upstream is under review22:56
*** loq_mac has quit IRC22:56
clarkbI see thanks22:56
clarkb(change has been approved)22:56
SpamapSclarkb: probably related to this though:
uvirtbotLaunchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New]22:57
SpamapSclarkb: I pasted the lines with data in the comments in hopes it helps :)22:57
clarkbSpamapS: it could be however that was a failure local to test slaves and had nothing to do with pypi.o.o22:58
*** rfolco_ has quit IRC22:58
clarkb(could be because I think those local test slaves use apache now)22:58
*** Ryan_Lane has quit IRC22:58
clarkbmordred: is it possible that your apache vhost needs updating?22:58
openstackgerritA change was merged to openstack-infra/config: Make python3.3 nonvoting for reviewstats.
*** bpokorny1 has quit IRC22:59
jeblairclarkb: what are you talking about?22:59
jeblairclarkb: what test slaves run apache?22:59
clarkbjeblair: the pbr integration test slaves serve the local mirror out of apache iirc22:59
lifelessclarkb: thanks23:00
clarkbthere was some reason mordred stopped using file:/// URIs for the local mirror. I would need to look at git logs23:00
*** rongze has quit IRC23:00
lifelessthe file:// codepaths are not the same as the http:// codepaths23:00
lifelesswe were not getting representative behaviour out of them23:00
mordredthat is correct. it turns out that pip does not do case insensitive searching on wheels via file:/// urs23:00
mordredbut yes. lifeless said it more succinctly23:00
*** sarob has joined #openstack-infra23:00
*** nati_ueno has joined #openstack-infra23:01
jeblairclarkb: okay, so what failure do you think is related to that?23:01
*** thomasem has quit IRC23:01
clarkbjeblair: SpamapS thought his pypi.o.o failures may be related to bug 1257905. I only think that is possible if the apache on pypi.o.o and on the test slaves is exhibiting a common problem23:02
uvirtbotLaunchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New]
clarkbbecause 1257905 is occuring without talking to pypi.o.o23:03
jeblairclarkb: i thought 1257905 was related to case mismatches, not connection errors.23:04
SpamapStrue, sounds like coincidence and not truly related23:04
SpamapS(I saw pbr problems in both and thought too much)23:04
clarkbjeblair: that a theory anyways. case mismatches don;t happen when wheels are not enabled (fwiw I don't think spamaps problem is related at all, I am just trying to sort out if they could be)23:05
jeblairi believe portante reported some pypi.o.o connection problems earlier.  perhaps there are rackspace network glitches.23:05
jeblairclarkb: but then, it may have been on a requirements job, so i guess all bets are off.23:05
portantejeblair: manifested as pbr install failures23:06
fungii wonder whether we wouldn't be better served by normalizing all requirements lists to match the capitalization used on pypi, so that file:// urls will "just work" to local trees23:06
SpamapSportante: same for me.23:06
fungi(and then enforcing that capitalization in requirements gating)23:07
*** _david_ has quit IRC23:08
*** rfolco_ has joined #openstack-infra23:10
jeblairclarkb: i've been unable to replicate the keypair leak on rax with either a successfull or failed image build23:11
jeblairclarkb: hopefully the new logging will help pin it down further23:11
clarkbjeblair: ok, thanks23:12
clarkbzaro: is that something we ran into with our jenkins?23:14
clarkbcurious to know if I approve it before harshar's testing will we have any idication it works?23:15
*** Mithrandir has quit IRC23:15
*** oubiwann_ has quit IRC23:15
*** datsun_F40PH has quit IRC23:16
*** mfer has quit IRC23:16
*** ArxCruz has quit IRC23:17
*** rfolco_ has quit IRC23:18
mordredfungi: well...23:18
mordredfunny you say that23:18
mordredfungi: global-requirements is supposed to be capitalization aligned23:18
pleia2clarkb: updated the infra-pubs-cleanup etherpad with the last commit before delete everything for the new branches in gerrit (no rush, whenever you have some time)23:19
mordredbut that doesn't mean that transitives are23:19
fungioh! very good point23:19
fungiand sad23:19
clarkbpleia2: ok, I am trying to get through code review now. Can poke at that after23:19
*** Mithrandir has joined #openstack-infra23:19
*** rfolco_ has joined #openstack-infra23:20
fungimordred: so basically, pip's use of local filesystem urls is broken except for very small/carefully constructed dependency chains?23:20
jeblairyay, zuul is doing an 8x merge23:20
*** vipul is now known as vipul-away23:21
fungimordred: was there a suggestion that pip 1.5 (2.0?) is going to fix that by analyzing local file trees and doing similar case/symbol munging to match package names?23:21
* fungi has some vague memory that got mentioned23:21
jeblairclarkb: i cleaned up rax keypairs23:21
fungijeblair: clarkb: over the weekend i did the same for the rax openstackci account23:22
fungisince i hit the keypair quota there23:22
clarkbcould the bug be in novacleint if we are seeing it in places nodepool doesn't touch?23:22
jeblairfungi: oh fun!23:22
fungiabout half were from very old devstack slaves and the other half seemed to be from aborted runs of the script23:23
jeblairclarkb: launch-node is pretty close to nodepool in behavior here23:23
*** gyee has joined #openstack-infra23:23
*** michchap has joined #openstack-infra23:24
openstackgerritA change was merged to openstack-infra/reviewstats: Fix pep8 tests
jeblairwith the logging in place, we'll be able to determine whether nodepool is actually requesting the deletion the next time we see a leaked key23:24
clarkbsdague: mtreinish: is something we should WIP until it can pass tests regularly?23:25
*** pabelanger_ has quit IRC23:26
*** pabelanger has joined #openstack-infra23:26
*** esker has joined #openstack-infra23:27
*** rfolco_ has quit IRC23:27
zaroclarkb: about 58140, no we didn't run into that because i don't think we have that setting enabled on any of our slaves.23:27
clarkbzaro: ok thanks. I will wait for hashar to respond then23:28
*** dkliban_ has quit IRC23:28
*** rossella_s has quit IRC23:29
*** esker has quit IRC23:32
clarkbI just approved it had a bunch of +2s but no approval (looked safe to me)23:32
*** rfolco_ has joined #openstack-infra23:32
*** weshay has quit IRC23:36
*** Mithrandir has quit IRC23:37
*** Mithrandir has joined #openstack-infra23:37
*** ryanpetrello has joined #openstack-infra23:37
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: ModuleRegistry now caches entry points lookup
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Don't assume substitation for a non-template job
*** rfolco_ has quit IRC23:38
russellblifeless: clarkb ^  what we were just discussing in -dev23:41
lifelessrussellb: already voted with my puny +1.23:41
*** vipul-away is now known as vipul23:41
clarkbrussellb: and it has my +2 too, I will give other cores a chance to review it though that change is simple enough  Iwill just approve in a bit23:41
russellbthanks guys23:42
*** Ryan_Lane has joined #openstack-infra23:42
*** rfolco_ has joined #openstack-infra23:42
*** paul-- has joined #openstack-infra23:42
clarkbjeblair: is ready to have your -2 removed23:44
* fungi disappears again to head home. bbiab23:45
*** Ryan_Lane has quit IRC23:46
openstackgerritA change was merged to openstack-infra/config: Add ironic puppet module to stackforge
openstackgerritA change was merged to openstack-infra/config: Don't assume substitation for a non-template job
openstackgerritA change was merged to openstack-infra/config: Changing the output directory that gets picked up for the API Quick Start
*** salv-orlando has quit IRC23:49
*** yamahata_ has joined #openstack-infra23:50
openstackgerritlifeless proposed a change to openstack-infra/reviewstats: Use six to handle cpickle.
*** mrodden has quit IRC23:50
openstackgerritRussell Bryant proposed a change to openstack-infra/reviewstats: Add --csv-rows option
*** ryanpetrello has quit IRC23:51
clarkbsdague: re maybe tomorrow we can sort out how to land that change and its siblings? I am happy to babysit that23:51
*** thedodd has quit IRC23:52
*** mriedem has quit IRC23:52
*** loq_mac has joined #openstack-infra23:54
openstackgerritA change was merged to openstack-infra/reviewstats: Remove arata and devananda from TripleO core.
*** rongze has joined #openstack-infra23:56

Generated by 2.14.0 by Marius Gedminas - find it at!