Tuesday, 2014-04-08

*** thuc has joined #openstack-infra00:00
*** thuc_ has joined #openstack-infra00:01
*** mrda has quit IRC00:01
*** thuc_ has quit IRC00:02
harlowjahmmm, did someone kill wiki.openstack.org?00:02
*** thuc_ has joined #openstack-infra00:02
harlowja(Cannot contact the database server) :-/00:02
fungiharlowja: mysql update--apache probably needs a restart there00:03
*** thuc_ has quit IRC00:03
*** thuc_ has joined #openstack-infra00:03
openstackgerritDoug Hellmann proposed a change to openstack-infra/config: Add cross-project unit tests for oslo libraries  https://review.openstack.org/8548700:03
*** mrda has joined #openstack-infra00:03
*** timrc is now known as timrc-afk00:04
dhellmannfungi, jeblair : that version includes a tool for checking the configuration ^^ (it shows that I need to add more jobs)00:04
*** thuc_ has quit IRC00:04
*** thuc has quit IRC00:04
*** thuc has joined #openstack-infra00:05
*** jergerber has joined #openstack-infra00:09
*** jergerber has quit IRC00:09
*** nati_uen_ has joined #openstack-infra00:11
*** nati_ueno has quit IRC00:12
*** nati_ueno has joined #openstack-infra00:12
*** UtahDave has quit IRC00:15
*** nati_uen_ has quit IRC00:16
*** msabramo has joined #openstack-infra00:18
*** SumitNaiksatam has quit IRC00:18
jeblair#status alert All of the project infrastructure hosts are being restarted for security updates.00:20
openstackstatusjeblair: sending alert00:20
clarkbI guess that is my go ahead?00:20
* clarkb reboots logstash-worker nodes00:21
openstackstatusNOTICE: All of the project infrastructure hosts are being restarted for security updates.00:21
*** ChanServ changes topic to "All of the project infrastructure hosts are being restarted for security updates."00:21
fungishould i take care in rebooting the gerrit and jenkins servers? we need to put zuul on hold first?00:21
openstackstatusjeblair: finished sending alert00:22
jeblairfungi: how about i save the zuul queue status00:22
*** weshay has joined #openstack-infra00:22
jeblairmordred: please don't reboot eavesdrop00:22
jeblairfungi: done, all clear00:23
fungirebooting the world00:24
*** openstackgerrit has quit IRC00:24
mordredjeblair: kk00:26
clarkbelasticsearch cluster is yellow00:27
clarkband recovering00:27
funginote, regenerating the gerrit ssh api/git host key will be potentially disruptive for all devs00:28
fungiwe might want to consider that one an acceptible risk00:28
clarkball worker nodes were kicked and had A-C worker processes started00:28
mordredfungi: noted00:28
*** amotoki has joined #openstack-infra00:29
*** nosnos has joined #openstack-infra00:29
clarkbI am not seeing any jobs queue on logstash.o.o but I am not expecting taht to be a real issue right now (we must be waiting for jenkins to do stuff)00:29
jeblairfungi: is gerrit back up?00:30
fungijenkins07 is back up and responding... and that was the last one i rebooted so checking the others now00:30
jeblair(asking if i should re-enqueue the zuul changes now)00:30
fungigerrit does not seem to be yet00:30
fungichecking it's tonsils now00:30
fungissh: connect to host review port 22: Connection refused00:30
fungiquick, to the rackspace00:30
clarkbis it fscking?00:30
fungientirely likely00:31
jeblairi had 2 nodes also not come back up yet, the others did00:31
mordredfungi: I've got the same for graphite and groups00:31
*** mbacchi has quit IRC00:31
mordredand cacti00:31
mordredthe others are rebooted00:31
jeblairoh, one of them just came back up, so yeah, fsck seems likely00:31
NobodyCamI assume review.openstack.org is rebooting too?00:32
fungii think a recent browser update must have knocked out my ability to use the web-based virtual console in the rackspace dashboard00:32
NobodyCamoh Ty00:32
*** aconrad has quit IRC00:32
fungi"Could not connect to console in the expected amount of time. Please refresh the page or try again later."00:32
jeblairfungi: nova console-log or whatever it's called work?00:32
fungifinding out!00:33
*** khyati_ has quit IRC00:34
fungiERROR: There is no such action: os-getConsoleOutput (HTTP 400) (Request-ID: req-35fd729c-7978-4c3a-ac23-fff6848ea7f0)00:34
fungimust not be supported?00:34
*** blamar has quit IRC00:34
jeblairhrm.  it is an old server00:34
*** blamar has joined #openstack-infra00:34
jeblairi'll try the web console00:35
clarkblet me know if I should check anything00:35
fungiplease do. still refusing ssh and flying blind as to why now00:35
mordredall up execpt for graphite now00:36
*** timrc-afk is now known as timrc00:36
fungii'm guessing fsck picked up an error and is prompting to enter single-user mode to repair it00:36
jeblairfungi: i think it has a really big filesystem...00:36
mordredgraphite is now back up00:37
fungimordred: please re-stop apache on graphite if you haven't00:37
jeblairProxy reports "500 Internal Server Error"00:37
jeblairfungi: why is apache stopped on graphite?00:37
fungioh, graphite00:37
fungimordred: sorry about that. meant cacti00:38
mordredstopped on cacti00:38
mordredrestated on graphite00:38
fungitoo many graphing things00:38
jeblairfungi: review is up00:38
fungireview.o.o finally came up00:39
fungiand is responding on webui00:39
fungi00:39:27 up 14 min00:39
fungioh, right, spent that long running fsck00:39
fungithat's time since kernel started, not time since runlevel change00:40
fungijenkins.o.o is still down too00:40
mordredas we rekey on things and re-enroll in puppet  - should we enroll in the salt master too? (two birds, one stone?)00:41
mordredand/or - I bet we should re-CA salt master?00:41
*** thuc has quit IRC00:41
*** thuc has joined #openstack-infra00:42
clarkbmordred: I don't think we should mix things00:42
fungiall the jenkins masters except jenkins.o.o are up and responding again00:42
mordredclarkb: k00:43
fungijenkins.o.o acts like it's still running a fsck too00:43
mordredclarkb: we should probably re-CA the salt master though and re-kick the things that are connected00:43
*** alexpilotti has quit IRC00:44
*** arosen has quit IRC00:44
fungipotentially good news, a friend says that centos 6.4 runs too old of an openssl to be affected00:44
clarkbfungi: we don't use that version though00:44
clarkbI will confirm00:44
*** thuc has quit IRC00:46
fungisince we're on 6.5 now00:48
dstufftcentos/rhel finally updated their OpenSSL, just in time for heartbleed00:49
fungijenkins.o.o is *still* not back up00:49
fungidstufft: just wondering when the glacier of centos security updates will catch up to this one00:49
dstufftfungi: I'm in #centos at the moment, we're testing a openssl build on warehouse.python.org I think00:50
dstuffternest is doing it not me, I'm just observing00:50
jeblairall my hosts are up now00:50
dstufft(ernest is one of the python.org infra team members w/ me)00:50
fungidstufft: oh, awesome! maybe things are faster there than they used to be00:51
clarkbif centos updates I can take that on, and try to do it with as little impact as possible00:51
dstufftPyPI runs on CentOS00:51
dstufftSo I have an invested interest :]00:51
fungiall my hosts are back online *except* jenkins.o.o (next-gen vm but not performance flavor)00:52
clarkbfungi: jeblair: looking into puppet CA stuff now. I think we should kill CA, create new certs for all nodes then copy those out to our nodes00:52
fungistill refusing connections, which makes me suspect fsck is running00:53
funginova console-log doesn't work on it either00:53
jeblairfungi: i think it is also a big old server00:54
fungi30gb standard instance, yep00:54
fungisame as review.o.o00:55
mordredclarkb: I support that plan00:55
mattoliveraufungi: can you access the console via the web ui?00:55
clarkbblkperl wrote a blog http://blkperl.github.io/replace-puppet-ca.html00:55
clarkbblkperl: nibalizer whic process is more correct?00:55
phschwartzDo we have a good guide to setting up and working with zuul00:55
jeblairphschwartz: now isn't a good time; we're dealing with the openssl issue00:56
fungimattoliverau: recently i've been having no luck wit the web comsole--i think something changed with my browser to break it00:56
nibalizerya blkperls blog00:56
fungimattoliverau: icedtea plugin used to deal with it fine00:56
nibalizerthe puppetlabs wiki is super out of date00:56
nibalizerare you replacing CA?00:56
phschwartzjeblair: sorry I can wait. Just connected to my bot and should have read.00:56
nibalizeror just revoking all the things?00:56
fungimattoliverau: but i get connection timeouts after the little progress indicator spins for a while00:57
*** yamahata has joined #openstack-infra00:57
mattoliveraufungi: you might have to.. *shudder* install sun/oracle java plugin.00:57
clarkbnibalizer: I think we should go ahead and replace CA00:57
nibalizerclarkb: i agree00:57
nibalizer(as you already know)00:57
fungimattoliverau: yeah, i don't think i have anywhere safe enough to install that these days00:58
nibalizeryou can skip a ton of the steps on blkperls blog because you don't have a separate CA server00:58
jeblairmattoliverau: yeah, i won't be doing that either :(00:58
mordredclarkb: before we kill the old one - puppet cert list -a  | grep '^\+' | awk '{print $2}' | sed 's/"//g' - will give us a list of the current hosts00:58
mordredthat we've accepted00:58
clarkbmordred: can you put that list in /root somewere?00:58
nibalizeryou have shared master/CA server so you'll bea ble to nuke the /var/lib/puppet/ssl dirs on everything, run puppet master on the server, then restart apache on the master, and start running puppet agent -t00:58
jeblairmattoliverau: fortunately, i _think_ the nova console log command works with newer hosts, but these are old ones00:58
fungimattoliverau: "Could not connect to console in the expected amount of time. Please refresh the page or try again later."00:58
mordredclarkb: it is now in /root/allcerts.txt00:59
jeblairmattoliverau: i got an error 500 from the proxy00:59
clarkbnibalizer: mordred: then I figure we moev /var/lib/puppet/ssl aside just incase we wedge outselves00:59
clarkbthen delete it next week or something00:59
mordredclarkb: ++00:59
nibalizerclarkb: sounds good01:00
mattoliveraujeblair: so is the console not up at all.. that may mean it isn't even fscking01:00
jeblairi restarted nodepool again because it wasn't emitting statsd01:00
*** timrc is now known as timrc-afk01:00
mattoliveraufungi: ^^01:00
jeblairmattoliverau: we saw this on earlier hosts that eventually came up01:00
fungimattoliverau: well, review.o.o did the same thing, and eventually came up (starting with an uptime of ~15 minutes, which suggests it was fsck taking that long to complete)01:01
mattoliveraujeblair: awesome, what good is a console, that you can't access01:01
*** gyee has quit IRC01:01
mattoliveraufungi: yeah, sounds like it. would be nice to get a console to see it checking tho :(01:01
jeblairor answer any questions it may have...01:01
jeblairfungi: have you tried nova console log with jenkins?  it's probably the same vintage so it probably won't work... but still worth a shot?01:02
fungijeblair: i did, first in fact, before trying the web comsole01:02
mattoliveraufungi: is this a rackspace cloud server? If so, maybe I should find someone to poke and say the console is teh broken.01:02
clarkbmordred: I am updating etherpad with puppet plan can you look at it and fill in holes?01:03
mordredclarkb: yah01:03
jeblairfungi, mattoliverau: jenkins.o.o is answering pings, so it's _probably_ fscking.  the idea that it's waiting at a prompt is worrying though.01:03
*** yaguang has joined #openstack-infra01:03
phschwartzmattoliverau: If it is I can poke it01:03
fungimattoliverau: i can't rule out the possibility that it's my browser, though now it's getting further (i actually see "status: connecting to dfw.servers.console.rackspacecloud.com, port 443..." indefinitely when i try01:03
jeblairphschwartz, mattoliverau: it is01:04
jeblairfungi: i saw that for a long time before i hit the proxy error01:04
phschwartzwould you like me to take a look at the logs for the HV and the console?01:04
jeblairphschwartz: i think that would be great, fungi?01:04
fungiphschwartz: if you don't mind, that would be awesome... this is jenkins.openstack.org 5d35421b-95aa-4e55-adf0-3a1a3ea8dca201:05
mordredclarkb: what if ...01:05
mattoliveraujeblair: ok, so the networking is up, so obviously as at least mounted the / disk so it can get it's ip configuration.01:05
fungijeblair: yep, now i'm getting the proxy error rather than the timeout i was getting earlier01:05
fungi'network error: proxy reports "500 internal server error"'01:06
fungiso i guess previously the web client was hitting timeouts reaching the proxy01:06
*** blamar has quit IRC01:07
phschwartzfungi: what data center is it in?01:07
fungiphschwartz: dfw01:07
*** aconrad has joined #openstack-infra01:08
phschwartzfungi: Not reporting any errors, let me see if I can get the console up01:09
fungiyeah, the rackspace web dashboard seemed to think that server was just fine01:09
*** sarob has quit IRC01:10
phschwartzfungi: The hv thinks it is too.01:10
*** sarob has joined #openstack-infra01:10
phschwartzfungi: I can't get the console up. Going to try and get one of our cloud server ops to take a look01:11
fungithanks phschwartz01:11
*** blamar has joined #openstack-infra01:11
*** Sukhdev has quit IRC01:13
clarkbmordred: why don't we start the puppet process fi no one needs puppet for stuff right now01:14
clarkbmordred: then we can work out entire process on a couple nodes01:14
*** sarob has quit IRC01:15
fungiclarkb: that seems reasonable01:15
fungiphschwartz: i can get to the console now01:15
fungiand the server's miraculously up01:15
mattoliveraufungi: what01:15
mattoliverauwhats the uptime01:15
mattoliveraufungi: ^01:15
fungi01:15:49 up 50 min01:15
mordredjeblair: clarkb and I are haing a disagreement :)01:16
phschwartzfungi: hmm, maybe the HV was having an issue. But I don't see any nagios reports of issues.01:16
fungii think it was fsck'ing that whole time01:16
*** nosnos has quit IRC01:16
mattoliveraumust've been, and you can't get a console while it's happening :(01:16
phschwartzfungi: that sounds like a reasonable assumption01:16
*** atiwari has quit IRC01:16
mordredjeblair: we may need you to weigh in and break the tie - and I need to go afk my ride is here - back online soon01:16
phschwartzmattoliverau: I should have been able to get a console so I am going to have an admin do a once over of the hvm just incase.01:17
fungiphschwartz: yeah, it was responding to icmp echo request, but refusing connections on port 2201:17
phschwartzdefn sounds like single user mode.01:17
fungiwhich leads me to think it was in secure runlevel the whole time01:17
*** blamar has quit IRC01:17
fungiso apparently the web console does actually work in my browser, just not at the times when it would actually be useful to me ;)01:18
fungiclarkb: so what is the meat of the disagreement?01:18
mattoliveraufungi: yeah, so it's a console fail01:18
*** jeblair has quit IRC01:18
clarkbfungi: if we remove existing puppet CA how do we regenerate certs for all the nodes. Do we generate on master then push to all nodes or have each node call home and regenerate certs that way01:18
clarkbfungi: I am arguing for having the master do it so that launch works without needing to copy all of the files back to the master later01:19
clarkbfungi: but that requires ssh -A to do file copy onto nodes01:19
fungiclarkb: if the clients can call home, they need their local key material replaced first before they submit a signing request to the master01:19
clarkbfungi: correct01:19
clarkbthey actually need it replaced in both cases01:19
mattoliverauphschwartz: it would be an interesting test to see if the console is always unavailable during a fsck or while in single user mode. tho that sounds stupidly wrong. Maybe getting an admin to take a look is a good idea.01:20
clarkbprocess would be something like, do master CA, for each agent stop agent, delete existing certs, either call home or copy in new data, start agent01:20
fungisure, but in the generate-on-master case the keys themselves would be generated on the master and pushed over the network rather than generated on the agent side01:20
clarkbfungi: correct01:20
*** aconrad has quit IRC01:20
mattoliverauphschwartz: And it's a good thing your around as you have access to see these things. Rackspace doesn't give me that kind of access since I left the UK.01:21
fungiin a pki scenario it's generally preferred to generate the key on the client since the mastyer doesn't actually need a copy. on the other hand when replacing servers we've abused the key copy on the master before01:21
clarkbfungi: we need a copy in this case01:21
*** timrc-afk is now known as timrc01:21
fungiphschwartz: yeah, thanks a bunch for looking into it!01:22
fungiclarkb: oh, does the server generate the cert from the key rather than from a csr?01:22
clarkbfungi: no, it will work the way you describe but then all of our launch node automation will break01:22
fungiin which case puppet sort of missed the point of separating keys and certs01:22
fungioh, our automation01:22
fungiwell, yes, it does assume generating keys on the master01:22
phschwartzmattoliverau: I am a racker on an Openstack dev team so I have the access, but getting an admin involved is a good step here.01:23
fungimainly because they're generated before the agent exists01:23
clarkbfungi: right bceause we need to presign01:23
clarkbotherwise initial boot breaks01:23
*** jeblair has joined #openstack-infra01:23
phschwartzI am trying to get much more involved with infra for the future so I will be around a lot to look into things and will make sure there is a way for people to reach me if needed.01:23
fungias a side note, i wonder whether it would be at all trivial to correct that, but it's a debate for later01:23
jeblairsorry i dropped out01:24
mattoliverauphschwartz: I'm a remote racker in Australia working on upstream openstack. I don't get access, I guess cause I'm upstream.01:24
fungiphschwartz: thanks for having an interest in our infrastructure in particular! i'll be thrilled to give you pointers to all the fun bits when we're not in the middle of a firefight01:24
phschwartzmattoliverau: hmm, you should have it01:25
fungiclarkb: so i agree. given the current situation with the launch automation, the distinction is academic for now01:25
clarkbjeblair: trying to sort out the best way to recert all of our puppet agents01:25
phschwartzmattoliverau: and were you at rax.io?01:25
clarkbjeblair: and didn't want tostart that surgery until we all agree on a course of action there01:25
mattoliverauphschwartz: I was indeed, I was the only openstack dev from Australia whom was there :)01:25
fungiclarkb: so i say we emulate what the current launch script does, unless it's sufficiently easier not to01:26
phschwartzmattoliverau: did you stay at aloft?01:26
mattoliverauI did :)01:26
clarkbfungi: the problem is we don't have access from master to agent via ssh at this point01:26
phschwartzmattoliverau: ok, I remember you now. lol01:26
clarkbwe only have that during launch01:26
clarkbso we would need to ssh -A or similar01:26
clarkbit is all pretty ugly01:26
fungiclarkb: right, though one of us can grant ourselves read access to the keys and ferry them via ssh01:27
mattoliverauphschwartz: lol, nice :)01:27
jeblairwhat was mordred arguing?01:27
clarkbjeblair: mordred wanted to use puppet agent --test on the nodes to have them regenerate cert on the node side, then sign on the master side01:27
clarkbjeblair: which would work fine except launch node scripts expect certs to exist on the masters01:27
*** julim has quit IRC01:28
*** signed8bit has quit IRC01:28
clarkbalso puppetdb is going to be a special case01:28
fungilarkb: as in 'ssh ci-puppetmaster cat /path/to/key | ssh wherever "cat > key"' and then do a local copy and delete on te destination host01:28
*** matsuhashi has joined #openstack-infra01:28
clarkbfungi: ya that would work too01:29
jeblairfungi, clarkb: i like that01:30
fungiprobably easier than trying to set up some temporary ssh push access to the agents after there is no master relationship01:30
clarkbok let me update etherpad with that then we can start doing this01:31
jeblairi'm going to reset the status and then reboot eavesdrop01:31
jeblair#status ok All services should be back up01:32
openstackstatusjeblair: sending ok01:32
*** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/"01:32
fungiclarkb: i did something similar in section 4 of https://etherpad.openstack.org/p/git-server-rebuild to preserve ssh host keys between server replacementd01:34
fungipiping through sudo like that you can avoid the local copy/delete cycle on the destination too, possibly01:35
*** SumitNaiksatam has joined #openstack-infra01:35
jeblairi think statusbot is done, but it hit another rate limit on the channel announcements01:36
jeblairi think it got the topics though01:36
*** openstack has joined #openstack-infra01:48
jeblairthe irc bots are all running01:48
clarkbfungi: I think we can do an explicit generate01:48
fungii guess 'puppet cert --generate ci-puppetmaster.openstack.org' will just do what's needed for this?01:49
clarkbfungi: yup it should01:49
fungilgtm then01:49
clarkbI have put #'s in front of nodes that in allcerts.txt that we shouldn't regenerate certs for01:49
clarkbhaving more eyes on that list to make sure I didn't exclude live nodes or miss dead nodes would be great01:49
jeblairclarkb: remove ask.o.o01:50
fungiyeah, we don't have ssh access to ask01:51
* clarkb takes a short break before breaking everything01:51
jeblairremove centos6.slave.openstack.org ci-master.openstack.org citest.openstack.org devstack-launch*01:51
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow 'Build on the same node' in trigger-builds builder  https://review.openstack.org/8566801:52
jeblairremove jenkins2.openstack.org large1.openstack.org large2.openstack.org01:53
jeblairremove precise.slave.openstack.org01:54
fungireview2, review.paas.hpcloud.net, review-dev201:54
jeblairprecise3k-1.slave.openstack.org precisepy3k*01:54
fungipuppet-dashboard, puppet-bootstrap01:54
fungii think we've met from both ends now01:55
fungiohm right, that did just get deleted recently01:55
jeblairwhich is actually translate-dev.o.o...01:56
*** bhuvan has quit IRC01:57
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Support Matrix Tie Parent Plugin in wrappers  https://review.openstack.org/8564201:58
clarkbok all those have been removed01:58
fungipypi and pypi.slave01:59
fungino, not pypi.slave01:59
fungijust pypi01:59
*** changbl has joined #openstack-infra02:00
clarkbisn't pypi.o.o a thing?02:00
clarkbhrm maybe not02:00
fungiit moved to static a while back and got deleted02:00
fungipypi.slave is definitely still a thing02:00
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Fix some minor typos in the trigger documentation  https://review.openstack.org/8588502:00
fungipypi is just a vhost on static now though02:01
*** wenlock has joined #openstack-infra02:01
fungiso that's 73 systems02:01
clarkbfungi: jeblair: mordred: if I start this now will you guys be around to help out with the re certing?02:01
clarkbI definitely don't want to do it all in one giant automated script that will break and need redoing02:01
fungiclarkb: i'll try to help out, but i'm rapidly running out of steam (nearing bedtime)02:02
clarkbat least not for the first few02:02
*** david-lyle has joined #openstack-infra02:02
clarkbwell I am going to stop apache on the master now02:02
clarkbok puppet master is now using a new cert02:04
*** sandywalsh has quit IRC02:05
clarkbgah puppetdb is cranky02:06
clarkbgoing to try fixing that by hand out of bad02:06
clarkb*out of band02:06
*** alff has quit IRC02:06
*** alff_ has quit IRC02:06
blkperlclarkb: theres a script that should fix that02:06
openstackgerritDavid Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Minor fixes in the installation documentation  https://review.openstack.org/8588602:07
blkperlclarkb: http://docs.puppetlabs.com/puppetdb/latest/puppetdb-faq.html#puppetdb-is-complaining-about-a-truststore-or-keystore-file-what-do-i-do02:07
clarkbright I think yo ucovered that in your blog02:08
clarkb Iam doing that now02:08
clarkbthat didn't fix it02:09
clarkbtehre were warnings about /var/lib/puppetdb not matcing certs in /var/lib/puppet wonder if it nooped in that case ...02:10
clarkbI hate puppetdb right now02:10
clarkbwhy wouldn't tis be like any other agent?02:10
clarkband why does puppetdb break the master?02:10
*** mattoliverau has quit IRC02:10
clarkbthis is far too tightly coupled02:10
*** sweston has quit IRC02:11
clarkbwtf it didn't create a /var/lib/puppet/ssl02:11
clarkboh its in /etc/puppetd02:12
fungiclarkb: oh, that could be why the rekeying instructions did find -exec rm instead of rm -fr02:12
clarkbis that a symlink the warnings are about /var/lib/puppetdb02:12
clarkbfungi: no, I think this is just being stupid02:12
clarkbugh, and those depends on /var/lib/ssl there is a cyclic dependency02:14
clarkbso stupid02:14
clarkbI am going t omake puppetdb a special case and generate its stuff local to it02:14
* mgagne brings popcorn02:16
clarkbmgagne: :P02:17
clarkbhonestly though the chicken and egg here is not very amusing02:17
clarkbpuppet won't run without puppetdb, puppetdb won't run without certs02:17
mgagneclarkb: I know one of our team had this exact problem and they cursed at their monitors for days. Unfortunately, I don't have the exact info on how they fixed it02:18
mgagneclarkb: trying to find where they put their work :-/02:18
clarkbI am ready to uninstall puppetdb02:18
blkperlI don't think it needs to run, it just needs to make the correct certs02:19
jeblairclarkb: i'm semi-around02:19
jeblairclarkb: let me know if you have a task for me02:19
*** nati_ueno has quit IRC02:19
clarkbjeblair: well I have new certs in puppetdb:/var/lib/puppet/ssl that I generated on puppetdb and signed by and on puppet master.02:19
clarkbnow I am running the command to rebuild the certs in /etc/puppetdb/ssl and have restarted puppetdb but puppet agent --test still doesn't work02:20
nibalizerclarkb: move aside puppetdb.yaml on the msaster02:20
nibalizerand remove reports = puppetdb from puppet.conf on the master02:20
nibalizerand then puppet will run02:20
nibalizerthats 'disabling' puppetdb from the master perspective02:20
clarkbnibalizer: but I want working puppetdb or no puppetdb02:21
clarkbnot reporting to it is a bad middle ground02:21
jeblairclarkb: isn't that 'no puppetdb'?02:22
fungii assume he means just long enough to get the updated cert onto the puppetdb server02:22
nibalizeralso could puppet cert generate and copy over02:22
clarkbI already have the cert signed and on the puppetdb02:22
nibalizerwhats the problem you're having?02:22
clarkbError 400 on SERVER: Failed to submit 'replace facts' command for ci-puppetmaster.openstack.org to PuppetDB at puppetdb.openstack.org:808102:23
* clarkb is looking at network connectivity now02:23
*** mattoliverau has joined #openstack-infra02:23
nibalizerdo you have an old CRL on the puppetdb server?02:23
clarkbah connection refused02:23
nibalizerdid you get the stuff in /etc/puppetdb ?02:23
nibalizerdid a firewall turn on?02:23
clarkbnibalizer: no I moved /var/lib/puppet/ssl aside and /etc/puppetdb/ssl aside02:24
clarkb8081 is only listening on the 10 address is that expected?02:24
nibalizerwhat's in jetty.ini ?02:24
nibalizerthere should be a listen address in one of those files02:24
nibalizerthat address should be the hostname02:25
nibalizerand maybe the 10 address is higher in /etc/hosts than the real address?02:25
clarkbssl-host = puppetdb.openstack.org02:25
*** zz_gondoi is now known as gondoi02:25
nibalizerhost puppetdb.openstack.org on puppetdb?02:26
fungiyeah, didn't we discover that setting an empty listen address maybe caused it to listen on all interface addresses (including ipv6)?02:26
nibalizeroh yea that!02:26
nibalizerdo that!02:26
clarkbwhat do I do?02:26
nibalizerssl-host =
fungiin the configuration on the puppetdb server02:26
fungii thought that was in the puppet config template for that module now though02:26
nibalizerright but puppet isn't running to set that up02:27
nibalizerthis is so broken chinken-and-egg02:27
jeblairclarkb: yeah, jetty.ini.bak.1396923405 has that02:27
clarkbnibalizer: right, puppetlabs should maybe consider fixin that02:27
fungii wonder what regenerated the jetty.ini02:27
clarkbcould be confused puppet agents with the CA having broken?02:27
clarkbthat is scary because they should just do nothing02:28
*** dkliban has quit IRC02:28
fungii have seen puppet agent think it needed to fall back on a locally cached copy of its config before (and find some horribly outdated one which shouldn't have been there or something)02:28
fungii believe someone mentioned a way to turn that behavior off, but i don't recall the specifics02:29
*** changbl has quit IRC02:29
nibalizerfungi: probably the 'fixit' script that clarkb ran, or maybe a puppetdb package?02:29
nibalizerclarkb: you ran the puppetdb-ssl-fixplz script right?02:29
fungioh, maybe scripty-thingy did it02:29
clarkbnibalizer: yes02:30
clarkbthat may have done it too02:30
nibalizerfungi: ohai-plz-to-be-fixing --srsly02:30
*** Shrews has quit IRC02:30
clarkbpuppet agent is running again on puppet master and puppetdb02:30
*** harlowja is now known as harlowja_away02:30
clarkbI am going to test our process on logstash-worker16 for the remaining nodes02:30
clarkbthen when that is done I may ask jeblair and fungi to poke at a couple nodes so that everything looks good02:31
fungiif it helps, i did check out the ca cert and it seems to have used sane options when generated02:31
*** Shrews has joined #openstack-infra02:31
fungiso we won't, like, have it expire on us in 30 days or anything annoying like that02:31
*** amcrn has joined #openstack-infra02:32
jeblairclarkb: let me know when you need a hand02:34
nibalizerdefault is 5 years02:35
nibalizeranything else I can do to help?02:35
clarkbjeblair: I am editing the etherpad as I find problems02:35
clarkbjeblair: will let you know when I think it is ready for generate abuse02:35
clarkbnibalizer: nope, I think we are good now thanks02:35
mgagneclarkb: finally found what I was looking for. They inject the CA and signed certs beforehand using hiera.02:37
*** melwitt has quit IRC02:38
*** Ryan_Lane1 has quit IRC02:38
clarkbjeblair: fungi ok I think the steps there work02:39
clarkbI am going to start with my same set of nodes from the reboot list02:40
*** esker has joined #openstack-infra02:40
clarkbany chance you guys want to do the same with your lists?02:40
*** esker has quit IRC02:40
*** harlowja_away is now known as harlowja02:40
jeblairclarkb: will do02:40
jeblairclarkb: question in etherpad, just to make sure...02:41
jeblairclarkb: also, have you done the cert generate for everything?02:41
*** asettle is now known as alex-lunch02:42
clarkbjeblair: I have not done cert generate for everything you will need to do that before you touch your nodes02:43
*** dcramer_ has joined #openstack-infra02:43
clarkbI kinda want to compare the list we generated against what we end up with in the puppet ssl dir02:43
fungiclarkb: recommended way for making sure this is working? just watch puppet-dashboard?02:45
clarkbfungi: I ran puppet agent --test on the first node I did manually02:45
clarkbbut yes dashboard should show last report times02:46
clarkboh wait, we may not be applying the right perms in /var/lib/ssl02:47
clarkbI think preserving perms was mostly what we wanted but the user needs to be puppet isntead of root02:47
clarkbactually nevermind it seems to work fien with the perms from master02:48
jeblairclarkb, fungi: i just pasted in a python script i'm using02:50
*** amotoki has quit IRC02:53
*** amcrn has quit IRC02:53
fungiokay, my section are rekeyed02:54
jeblairfungi: want to start on mordred's?02:55
fungiyep, just checking puppetboard to make sure they checked in first02:55
fungioh, of course all the jenkins servers are complaining because i set the jenkins packages on them to held status (so they wouldn't auto-upgrade)02:56
clarkblogstash* are done02:58
clarkbdoing elasticsearch* now02:58
jeblairdone with my precise hosts; working on centos now02:59
fungiclarkb: you have "ssh node 'sudo /etc/init.d/puppet start'" in there, but didn't that previously cause issues with the mysql module? has that ceased being an issue now?03:01
clarkbelasticsearch* done03:01
clarkbfungi: it should be fine if using the init script iirc03:01
clarkbits only when run directly with sudo eg puppet agent that breaks03:01
fungioh, good to know03:02
*** alex-lunch is now known as asettle03:03
*** maxbit has joined #openstack-infra03:05
*** sweston has joined #openstack-infra03:05
clarkbjeblair: looks like centos 6 has it now too03:06
clarkbI am going to see if I can upgrade git05 now03:06
jeblairclarkb: oh? i just checked and it didn't say...03:06
clarkbunless people want me to do anything with pupept recerting03:06
clarkbjeblair: http://mirror.centos.org/centos/6.5/updates/x86_64/Packages/ has it03:07
clarkbrax may be masking it for us though03:07
fungiokay, got through rekeying all of mordred's section too03:07
fungii'll go ahead and un-hold jenkins on all the jenkins masters so puppet will stop complaining about the held status03:08
jeblairclarkb: yeah, the [updates] repo is a mirror03:08
jeblairclarkb: uncommenting the #mirrors line in that section should do it03:09
* clarkb learns yum03:10
jeblairi've completed the puppet stuff for all my hosts03:10
jeblairclarkb: uncomment that line, then 'yum update'03:11
* clarkb tries that on git0503:11
clarkbjeblair: fungi: do we want to try cycling git nodes in a non impactful way or just reboot them?03:11
jeblairclarkb: just reboot i think03:11
clarkbjeblair: uncomment the line in /etc/yum.repos.d/CentOS-Base?03:12
jeblairclarkb: yep03:12
fungiand after they're all done, gerrit replicate --all03:12
funginot important for the moment, but tripleo seems to have died again03:13
fungii'll give #tripleo a quick heads up03:13
jeblairfungi: it's possible that nodepool lost track of the few remaining nodes in the restart03:13
clarkbwell that seems to have just broken facter..03:13
jeblairclarkb: not if you didn't say "yes"03:13
fungijeblair: good point03:14
clarkbjeblair: ah ok, it bailed without asking me to say yes03:14
jeblairclarkb: those hosts are probably missing the exclude03:14
jeblairfungi: it should clear up after 8 hours, but can you check and maybe nodepool delete them if that's what it looks like?03:14
jeblairfungi: it should think there are a bunch of used nodes in that state for several hours03:15
*** zehicle has quit IRC03:15
clarkbjeblair: yup appears to be the case fixing03:15
jeblairclarkb: i've got git.o.o and the others in that section, you get gitNN, ok?03:16
clarkbok I can do gitNN03:16
clarkbjeblair: fwiw I still can't get it to update after the facter thing03:16
clarkbI think it is finding the fastest mirror03:16
clarkbrather than the mirror03:16
fungijeblair: yep, that was it--clearing those now03:16
jeblairclarkb: yeah, that seems to be the case03:17
* clarkb fixes facter03:17
jeblairclarkb: baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/03:19
jeblairclarkb: that should do it then03:20
clarkbthat did it03:20
clarkbworking through gitNN now without rebooting03:21
* mordred is back - sorry for the delay - anything left I can help on?03:21
*** thuc has joined #openstack-infra03:22
*** thuc_ has joined #openstack-infra03:22
clarkbmordred: maybe keep any eye on puppetboard to make sure that we recerted all the nodes03:23
clarkbmordred: and look at nodes that aren't reporting in the last 20 minutes03:23
*** changbl has joined #openstack-infra03:24
jeblairclarkb: i had to do 'yum clean all' a few times on one of them because it wasn't downloading the repo data03:24
mordredclarkb: kk03:24
jeblairi'm going to reboot the non-git nodes03:25
clarkbjeblair: ok03:25
*** thuc has quit IRC03:26
mordredall of the -dev servers seem non-responsive- is that a known thing?03:27
jeblairmordred: no03:27
clarkb16.el6_5.7 is the version we want03:27
jeblairclarkb: correct03:27
clarkbI am double checking that it is installed on the gitNN nodes03:27
clarkbjeblair: you did git.o.o right?03:27
jeblairrpm -q openssl03:28
mordredclarkb: if I find a server that does not seem to have a cert - do you havea  script/process to fix? I'm assuming make on the master and copy to the machine?03:28
jeblairmordred: give me the fqdn i'll do it quick03:28
mordredjeblair: groups.o.o03:28
clarkbgit01-git05 are ready03:28
fungii thought i did groups.o.o03:28
mordredjeblair: groups.openstack.org if you want cut and paste03:28
*** wchrisj has quit IRC03:29
jeblairmordred: err: Could not call generate: A Certificate already exists for groups.openstack.org03:29
*** matsuhashi has quit IRC03:29
fungibut yeah, still looks unreported03:29
*** Ryan_Lane has joined #openstack-infra03:29
fungialong with groups-dev and jenkins-dev03:29
*** wchrisj has joined #openstack-infra03:29
jeblairmordred: where is it missing a cert?03:29
clarkbjeblair: did the tar step get neglected?03:30
clarkbfungi: ^03:30
jeblairclarkb: want to just reboot all the git servers together?03:30
fungiclarkb: that's what i'm checking, but i don't think so03:30
*** blamar has joined #openstack-infra03:30
mordredjeblair: oh - sorry, I got a cert-related puppet error then when rechecking unreported nodes03:30
clarkbjeblair: yeah I think so03:30
mordredI get this: err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=ci-puppetmaster.openstack.org]03:30
jeblairclarkb: why don't you just do them all03:30
mordredon mirror2603:30
clarkbjeblair: I was going to for loop a thing that logged in a sudo reboot -h now03:30
*** zehicle has joined #openstack-infra03:30
jeblairclarkb: ++03:31
jeblairclarkb: (including git.o.o)03:31
clarkbjeblair: ok doing that now03:31
fungion groups.o.o /var/lib/puppet/ssl/{certs,private_keys,public_keys}/groups.openstack.org.pem are all there with today's date03:31
clarkber less the -h now03:31
clarkbtoo much shutdown -h now habits for personal machines03:32
clarkb`sudo reboot` is what I am doing03:32
clarkblol command reboot not found :/03:32
mordredalso - mirror26 is a centos node too - does it need the yum update you guys just did for git*?03:32
mordredand - zm01 doesn't have puppet installed03:32
clarkbok all 6 git* nodes are rebooting03:33
jeblairsha1sums of those files don't match the puppetmaster03:33
fungii'm going to try restarting the puppet agent again on groups.o.o in case i restarted it too quickly after removing/replacing certs03:33
jeblairmordred: i got mirror26, but it's recent03:33
mordredjeblair: ok. cool03:33
* mordred is going to install puppet on zm0103:34
jeblairmordred: can you find out why it's not installed03:34
mordredjeblair: sure!03:34
jeblairbecause clearly it was at some point03:34
*** gondoi is now known as zz_gondoi03:34
*** zz_gondoi is now known as gondoi03:34
fungihow weird. the keys indeed don't match but the certs do. i'll retransfer those03:34
clarkbrunning replicate --all on review.o.o now03:35
*** weshay has quit IRC03:35
*** blamar has quit IRC03:35
jeblairfungi: the timestamps on the keys on the host were 2 minutes later than the master03:35
*** blamar has joined #openstack-infra03:35
* clarkb tries to clone stuff03:36
mordredjeblair: only think I can see in the root history is the dist-upgrade steps03:36
mordred2014-04-07 23:36:54 remove puppet-common 2.7.25-1puppetlabs1 <none>03:36
jeblairfungi: i'm guessing something out of sequence caused puppet to make some local certs03:36
mordredfrom the log03:36
mordredI think dist-upgrade boned it03:36
jeblairi think zuul-dev may have the same prob03:36
mordredah - yup - it didn't get the facter pin before the distupgrade03:37
mordredit has facter 2.003:37
jeblairsorry, i thought i ran puppet agent --test everywhere03:37
clarkbI am able to clone openstack-infra/config from git.o.o via all 3 clone methods03:37
jeblairactually i'm sure of it, it's right there in my history03:37
clarkbso I think git replicas are good03:37
fungierr: Could not call generate: groups-dev.openstack.org already has a requested certificate; ignoring certificate request03:38
fungianybody know where those get stashed so i can delete it?03:38
clarkbfungi: /var/lib/puppet/ssl03:38
clarkbfungi: you might also need to puppet cert something something on the puppet master03:38
clarkbfungi: puppet cert list --all to see alisting03:38
jeblairfungi: oh i thought you were just going to recopy from the master?03:38
fungijeblair: i did for groups.o.o03:39
fungifor groups-dev there was a pending csrt03:39
jeblairfungi: ah ok03:39
fungicertificate_requests was the subpath i was hunting for03:39
jeblairmordred: yep, my fault, sorry03:39
mordredzm01 fixed-03:39
mordredand reporting03:39
jeblairmordred: and yeah, i did the same thing on zuul-dev too03:40
mordredjeblair: want me to fix? or you got it?03:40
jeblairmordred: can you?03:40
mordredjeblair: yup. on it03:40
jeblairyour fingers are already warmed up03:40
fungi'sudo puppet cert generate groups-dev.openstack.org' seems to always create a /var/lib/puppet/ssl/certificate_requests/groups-dev.openstack.org.pem and then complain that it exists03:40
fungivery, very weird03:41
fungithis is with puppet agent stopped on groups-dev as well03:41
clarkbfungi: is that on the master?03:41
fungiclarkb: yep03:41
clarkbyo uwant to generate the cert on the master03:41
fungiclarkb: yep03:41
clarkbfungi: does puppet cert list --all show a cert pending?03:41
fungithat's what i'm trying03:41
clarkbfungi: if it shows one pending it might be easiset to just sign that cert then copy it the other direction03:41
clarkbI will need to do this with the puppetdb certs tomorrow03:41
fungiclarkb: yeah, it's listed03:42
jeblairdon't sign it unless you've verified it's correct03:42
clarkbjeblair: ya that03:42
jeblairlots of random people send us csrs03:42
fungitrying to figure out how to delist it so i can generate a new one03:42
clarkbwhen I did puppetdb only it and puppet master were listed, was easy to check and sing :)03:42
jeblairfungi: 'puppet cert clean' ?03:42
fungierr: Could not call revoke: Could not find a serial number for groups-dev.openstack.org03:43
fungithat seems to only work on those which have already been signed03:43
fungichecking manpage for viable options03:43
*** dkliban has joined #openstack-infra03:44
*** chandan_kumar has quit IRC03:44
fungii wonder if there's an additional groups-dev out there somewhere trying to get noticed03:44
clarkbI am going to patch my local(ish) nodes now03:45
*** thuc has joined #openstack-infra03:45
*** blamar has quit IRC03:45
fungifound! also have to rm /var/lib/puppet/ssl/ca/requests/groups-dev.openstack.org.pem03:46
mordredjeblair: zuul-dev fixed03:46
jeblairmordred: cool thanks03:46
mordredjeblair: did you say you were dealing with cert issue on mirror26? or do you want me to look in to that?03:47
fungiokay, groups-dev all fixed up03:47
fungijenkins-dev seems to have never checked in03:47
jeblairmordred: i was not, but i will look at mirrir26 now03:47
*** thuc_ has quit IRC03:48
*** wenlock_ has joined #openstack-infra03:48
*** timrc is now known as timrc-afk03:48
*** thuc has quit IRC03:49
fungiokay, puppetboard shows groups and groups-dev rporting in now03:49
*** amcrn has joined #openstack-infra03:49
mordreddevstack-launch.slave.openstack.org is not - but I think it's a thing we don't care about, yeah?03:50
fungipuppet agent is unable to start on jenkins-dev03:50
jeblairmordred: i think it doesn't exist anymore, right?03:50
mordredit does exist03:50
*** chandan_kumar has joined #openstack-infra03:50
fungimordred: i think it still is pending deletion03:50
mordredbut I'm not sure why it still exists03:50
mordredfungi: should we fix it or delete it?03:50
fungimordred: i vote delete03:50
jeblairkill it03:50
* mordred on it03:50
fungiplease clear the record from dns too if you have time03:51
mordredfungi: while you're doing certs - you wanna delete it from the cert list?03:51
fungimordred: it should not be in the new cert list03:51
jeblairhrm, the ca.pem on mirror26 is old03:51
clarkbgah /boot doesn't have enough space on disk to mkinitramfs03:51
fungiclarkb: started from a fresh cert list03:51
clarkbmaybe ubuntu sould make /boot bigger by default03:51
jeblairbut the rest of the certs are there...03:51
clarkbjeblair: the rm /var/lib/puppet/ssl should've removed it then the first puppet run grab it again03:52
clarkbjeblair: you might want to run that node through the deletion coyp steps just to be safe03:52
mordredjeblair: puppetdb is still reporting it though03:52
jeblairit's because we ran puppet stop && rm -rf03:53
jeblairpuppet wasn't running so it short circuited03:53
clarkbjeblair: gah sorry03:53
clarkbmy  bug03:53
jeblair(puppet doesn't run agent on slaves)03:53
fungiCould not retrieve catalog from remote server: Error 400 on SERVER: Must pass mysql_root_password to Class[Openstack_project::Nodepool] at /opt/config/production/modules/openstack_project/manifests/nodepool.pp:3 on node jenkins-dev.openstack.org03:54
clarkbfungi: I saw that prior to all of this03:54
fungiso we have broken puppetry for jenkins-dev looks like03:54
clarkbfungi: so :/03:54
fungiwhich is why i can't start the puppet agent there03:54
jeblairwhy can't that be detected by the syntax checks?03:55
*** chandan_kumar has quit IRC03:56
* jeblair despairs of puppet03:56
clarkbok laptop is rebooting back in a bit03:56
clarkbfungi: any idea how to convince wheezy to upgrade to openssl*u5 on rax?03:56
clarkbfungi: I assume same issue there as with our precise nodes03:56
jeblairsomeone should just yank nodepool from jenkins-dev03:56
jeblairlet me know if you want me to do that03:56
jeblairmordred: mirror26 is running03:57
*** gondoi is now known as zz_gondoi03:57
clarkbjeblair: tehre is a change up to fix it on jenkins-dev03:57
fungiclarkb: probably, if you don't see security.debian.org in the sources.list03:57
mordredjeblair: awesome. I'm deleting dns for devstack-launch03:57
clarkbjeblair: would be nice to just get that thorugh03:57
jeblairclarkb: i've reviewed it 5 times, i'm not going to review it again right now :(03:57
jeblairclarkb: but if you want to pick this up tomorrow, we can probably just leave jenkins-dev unreporting03:58
mordredjeblair, fungi: dns record deleted for devstack-launch03:58
jeblairmordred: ack, thx03:58
*** wchrisj has quit IRC03:59
*** sabari has joined #openstack-infra04:00
jeblairi have an early flight tomorrow and have not packed; i should really go prepare for that04:01
mordredjeblair: ++04:01
fungijeblair: thanks for all the help--travel well!04:01
mordredfungi: for reference, stale nodes in pupeptboard can e removed with "puppet node deactivate devstack-launch.slave.openstack.org"04:01
jeblairi think we've gotten the high-prority stuff done at least04:01
jeblairmordred, clarkb, fungi: thanks and good night04:01
clarkbalso xubunty should really tell me what is broken when it wants to phone home04:02
clarkbI have no idea what is broken right now04:03
clarkbbut it insists on calling home without giving me any info04:03
clarkbpleia2: ^04:03
fungii'm going to get some sleep now, and stop caring about puppet on jenkins-dev for the moment. if anyone else cares about it working tonight, feel free to have a look04:03
mgagnejeblair: you could detect such "syntax errors" by adding unit tests. But I do not believe it to be possible with current openstack-infra setup as it heavily depends on secret data in hiera. This means unit tests (executed w/o hiera) would fail (due to missing hiera entries) while in production it would work as hiera is available.04:04
clarkbfungi: I will probably try to get it working properly tomorrow04:04
clarkbfungi: it should be fine in its current state04:04
fungiclarkb: it's probably just missing a variable passed in through its manifest or something04:04
mordredmgagne: I think we can set defaults in our hiera calls for what to use if there is no hiera04:04
mgagnemordred: true04:05
mordredclarkb: I'm reviewing khai's change to fix nodepool on jenkins-dev right now04:05
mgagnemordred: but missing required (no longer) secret values would now be harder to debug as puppet won't fail anymore and use invalid default value04:05
clarkbfungi: good night and thank you for staying up to help04:06
fungithe change up to fix nodepool on jenkins-dev looks only half done (for example, it only templatizes one of the leav prepare_node_*.sh scripts and adds the key var there, but leaves other prepare scripts out in the cold... might warrant happening in the bottom-level prepare_node.sh or something04:06
mordredmgagne: ++04:06
mgagnemordred: you can inject those in unit tests but you aren't testing against production config anymore so tests aren't so valid anymore04:06
* fungi places his wetware into standby for recharging04:07
*** Ryan_Lane has quit IRC04:07
clarkbhuh I think openvpn might be breaking me04:07
clarkbit is trying to start on boot but can't read a passwrd on non existant stdin04:07
clarkbwhich is lolzy04:08
pleia2clarkb: it should tell you if you click on the details thing (which should exist), but honestly I just turn apport off because I don't care when things crash and I don't like it when it sends reports from my day to day system04:08
clarkbpleia2: it just says apport thing04:08
clarkbpleia2: basically telling me apport is going to take my password and collect data as root04:08
clarkbwhich isn't super useful04:08
clarkbmaybe to people that don't grok linux but for me I want to see the error04:09
clarkbI think it is related to openvpn though04:09
*** maxbit has quit IRC04:11
clarkbwell that wasn't it. I see network manager is complaining about dnsmasq not existing though04:12
*** chrisstreeter_ has joined #openstack-infra04:12
clarkbthat is the only error from the most recent boot04:14
*** chandan_kumar has joined #openstack-infra04:14
*** sabari has quit IRC04:14
clarkbguessing that is a race?04:14
*** wchrisj_ has joined #openstack-infra04:15
pleia2pretty sure there is a bug out for that04:16
clarkbya looks like xfce starts dnsmasq but network manager starts at boot04:16
clarkboh nevermind network manager starts dnsmasq04:16
clarkbso network manager must be looking for it before it forks and execs it04:17
* clarkb ignores for now as that isn't too broken04:17
*** unicell has quit IRC04:18
*** kashyap` is now known as kashyap04:18
*** ianw has quit IRC04:19
*** wchrisj_ has quit IRC04:19
*** thuc has joined #openstack-infra04:19
*** ianw has joined #openstack-infra04:19
*** matsuhashi has joined #openstack-infra04:19
ianwdoes anyone know off-hand how to add color to jenkins comments?04:20
ianwsorry, gerrit, not jenkins04:20
clarkbianw: if you look in openstack-infra/config/modules/openstack_project/manifests/gerrit.pp you will see what we do to the comment links to color the jenkins results04:21
clarkbin gerrit04:21
clarkband my irc box is updated. I suppose I will have to reboot it too :( back in a bit04:21
*** clarkb has quit IRC04:22
zaroianw: i think gerrit comments support html markup.04:22
*** wchrisj_ has joined #openstack-infra04:23
*** clarkb has joined #openstack-infra04:25
*** sabari has joined #openstack-infra04:27
clarkbthat was fun04:27
clarkbI suppose I should kick back for now and get ready for tomorrow04:27
ianwclarkb: clearly i'm missing something about it https://review.openstack.org/#/c/50776/04:28
ianwclarkb: oh, hang on, i see, if i write in that format it will markup for me04:29
clarkbit escapes html so you have to do it in special format that gets formated for you04:30
*** markmcclain has joined #openstack-infra04:31
*** Ryan_Lane has joined #openstack-infra04:33
*** Ryan_Lane has joined #openstack-infra04:33
*** wchrisj_ has quit IRC04:33
*** Ryan_Lane1 has joined #openstack-infra04:36
*** Ryan_Lane has quit IRC04:36
*** Ryan_Lane1 is now known as Ryan_Lane04:36
*** Ryan_Lane has quit IRC04:36
*** Ryan_Lane has joined #openstack-infra04:36
*** yfried has quit IRC04:38
ianwclarkb: success!  or should I say SUCCESS (https://review.openstack.org/#/c/50776/) :)04:39
*** wchrisj has joined #openstack-infra04:39
*** wchrisj has quit IRC04:45
*** imcsk8_lap has quit IRC04:50
*** imcsk8_lap has joined #openstack-infra04:50
*** Ryan_Lane1 has joined #openstack-infra04:52
openstackgerritA change was merged to openstack-infra/storyboard: Fix for not loading middleware  https://review.openstack.org/8569304:52
*** nati_ueno has joined #openstack-infra04:53
*** thuc_ has joined #openstack-infra04:54
SergeyLukjanovclarkb, evening04:55
clarkbSergeyLukjanov: hello04:55
SergeyLukjanovclarkb, is there any agreement on non-merging nodepool patches now?04:55
*** wchrisj has joined #openstack-infra04:56
SergeyLukjanov(I've read scrollback, but probably missed something)04:56
clarkbSergeyLukjanov: it probably just needs someone to babysit, that said we are probably going to be swamped babysitting updates to deal with the openssl vulnerabilities04:57
*** thuc has quit IRC04:57
SergeyLukjanovclarkb, oh, yup, thanks04:57
*** thuc_ has quit IRC04:58
*** wchrisj has quit IRC05:00
*** timrc-afk is now known as timrc05:02
*** harlowja is now known as harlowja_away05:03
*** skraynev_afk is now known as skraynev05:03
*** mihgen has joined #openstack-infra05:04
*** dkliban has quit IRC05:06
SergeyLukjanovclarkb, re openssl, is it re https://www.openssl.org/news/secadv_20140407.txt?05:07
*** Ryan_Lane1 has quit IRC05:08
*** amotoki has joined #openstack-infra05:08
clarkbSergeyLukjanov: yes, so far today we updated openssl on all of our nodes and rebooted them to make sure nothing was sticking around using old openssl05:08
*** ildikov_ has quit IRC05:08
clarkbSergeyLukjanov: then we killed the puppetmaster CA and created a new oen, and recerted all puppet agents05:09
*** zhiyan_ is now known as zhiyan05:09
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix sftp access on CentOS machines  https://review.openstack.org/8592405:11
mordredclarkb: I just found the silliest thing ^^05:11
*** oomichi has joined #openstack-infra05:11
*** timrc is now known as timrc-afk05:12
*** chrisstreeter_ has quit IRC05:12
*** markmcclain has quit IRC05:13
mordredSergeyLukjanov: ^^ if you get a sec05:18
*** alff has joined #openstack-infra05:18
*** alff_ has joined #openstack-infra05:18
clarkbmordred: it failed linting I -1'd with comment on how to fix05:18
mordredclarkb: thanks. I have tested it for functionality, btw05:19
clarkboh good, you run that against one of the git* boxes?05:19
SergeyLukjanovmordred, /me looking05:19
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix sftp access on CentOS machines  https://review.openstack.org/8592405:19
mordredrocket ships fixed05:20
*** pcrews has quit IRC05:20
SergeyLukjanovmordred, +205:22
SergeyLukjanovmordred, /me not approving due to the probably need to babysit / check after merge05:22
mordredSergeyLukjanov: thanks. I'll babysit - I want it to work for something else05:25
*** saper has quit IRC05:27
*** saju_m has joined #openstack-infra05:28
*** Ryan_Lane1 has joined #openstack-infra05:28
openstackgerritA change was merged to openstack-infra/config: Fix sftp access on CentOS machines  https://review.openstack.org/8592405:30
*** yfried has joined #openstack-infra05:31
*** ildikov_ has joined #openstack-infra05:32
*** pcrews has joined #openstack-infra05:32
*** dolphm has quit IRC05:33
*** dolphm has joined #openstack-infra05:34
*** _nadya_ has joined #openstack-infra05:37
*** dolphm has quit IRC05:38
*** dolphm has joined #openstack-infra05:40
*** salv-orlando has joined #openstack-infra05:41
*** _nadya_ has quit IRC05:42
*** jlibosva has joined #openstack-infra05:44
*** salv-orlando has quit IRC05:45
*** maxbit has joined #openstack-infra05:49
*** Clabbe has joined #openstack-infra05:50
*** Ryan_Lane1 has quit IRC05:52
*** salv-orlando has joined #openstack-infra05:53
*** oomichi is now known as ken1ohmichi05:58
*** ominakov has joined #openstack-infra05:58
*** wenlock_ has quit IRC06:01
*** wenlock has quit IRC06:02
*** _nadya_ has joined #openstack-infra06:04
*** thuc has joined #openstack-infra06:04
*** unicell has joined #openstack-infra06:05
*** thuc has quit IRC06:09
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add example of comment formatting  https://review.openstack.org/8593006:10
*** saper has joined #openstack-infra06:12
mordredsomethign has gone strange on hosts with jeepyb06:16
mordredI have looked at it in person and can't tell what it's trying to do - but it looks like it's lost track of /opt/jeepyb beinga  git repo06:17
openstackgerritSteve Kowalik proposed a change to openstack-infra/config: Also clone os-cloud-config in prepare_tripleo.sh  https://review.openstack.org/8593206:18
openstackgerritThomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates  https://review.openstack.org/8376106:19
*** salv-orlando has quit IRC06:20
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add sphix deps to setup.py  https://review.openstack.org/8593306:21
StevenKianw: Sphix? :-)06:22
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add sphinx deps to setup.py  https://review.openstack.org/8593306:23
ianwStevenK: oops, sorry.  pretty sure the patch isn't right, but it's not really very clear how to build the docs if you want to test your change06:23
mordredianw: you want "tox -evenv python setup.py build_sphinx"06:23
ianwmordred: ah, ok, thanks06:28
ianwi was avoiding tox because it has a hard requirement on > 1.6 which isn't in f1906:28
*** denis_makogon has joined #openstack-infra06:29
*** _nadya_ has quit IRC06:32
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add note on ci.openstack.org source  https://review.openstack.org/8593306:33
*** zehicle has quit IRC06:35
*** zehicle_at_dell has quit IRC06:35
*** zehicle has joined #openstack-infra06:36
*** Sukhdev has joined #openstack-infra06:41
*** flaper87|afk is now known as flaper8706:44
*** jamielennox is now known as jamielennox|away06:48
*** Ryan_Lane1 has joined #openstack-infra06:50
*** dizquierdo has joined #openstack-infra07:00
*** reed has joined #openstack-infra07:00
*** yolanda has joined #openstack-infra07:00
*** andreykurilin_ has joined #openstack-infra07:02
*** maxbit has quit IRC07:11
*** saju_m has quit IRC07:17
*** Ryan_Lane1 has quit IRC07:18
*** jcoufal has joined #openstack-infra07:23
*** e0ne has joined #openstack-infra07:23
*** mrda is now known as mrda_away07:25
*** andreykurilin_ has quit IRC07:25
*** saju_m has joined #openstack-infra07:25
*** flaper87 is now known as flaper87|afk07:26
*** jgallard has joined #openstack-infra07:27
*** ttx has quit IRC07:32
*** ttx has joined #openstack-infra07:32
*** ttx has quit IRC07:32
*** ttx has joined #openstack-infra07:32
*** saju_m has quit IRC07:32
*** alff has quit IRC07:35
*** alff_ has quit IRC07:35
*** morganfainberg is now known as morganfainberg_Z07:36
*** Sukhdev has quit IRC07:37
*** jp_at_hp has joined #openstack-infra07:40
*** sabari has quit IRC07:43
*** reed_ has joined #openstack-infra07:43
*** reed has quit IRC07:46
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc  https://review.openstack.org/5208007:46
*** dutsmoc has joined #openstack-infra07:49
*** ttx has quit IRC07:50
*** ttx has joined #openstack-infra07:50
*** ttx has quit IRC07:50
*** ttx has joined #openstack-infra07:50
*** matsuhashi has quit IRC07:53
*** ttx has quit IRC07:53
*** ttx has joined #openstack-infra07:53
*** ttx has quit IRC07:53
*** ttx has joined #openstack-infra07:53
ttxyay, key rotation day07:54
*** matsuhashi has joined #openstack-infra07:54
*** hashar has joined #openstack-infra07:55
*** e0ne has quit IRC07:55
*** andreaf has joined #openstack-infra07:56
*** e0ne has joined #openstack-infra07:56
*** saju_m has joined #openstack-infra07:58
*** ociuhandu has quit IRC07:59
*** e0ne has quit IRC08:00
*** derekh has joined #openstack-infra08:02
*** saju_m has quit IRC08:03
*** Ryan_Lane1 has joined #openstack-infra08:04
*** Ryan_Lane1 has quit IRC08:04
*** saschpe has joined #openstack-infra08:04
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Missing docstrings added  https://review.openstack.org/8571808:06
*** jpich has joined #openstack-infra08:07
*** mihgen has quit IRC08:09
*** Ryan_Lane has quit IRC08:09
*** nati_ueno has quit IRC08:11
*** vponomaryov has joined #openstack-infra08:14
*** saju_m has joined #openstack-infra08:15
*** chandan_kumar has quit IRC08:15
SergeyLukjanovttx, yup, or several days :)08:17
*** nati_ueno has joined #openstack-infra08:18
*** nati_ueno has quit IRC08:18
*** sweston has quit IRC08:19
*** sweston has joined #openstack-infra08:20
*** comstud has quit IRC08:20
*** dutsmoc is now known as comstud08:20
*** che-arne has quit IRC08:21
*** ihrachyshka has joined #openstack-infra08:24
openstackgerritA change was merged to openstack-infra/config: Make transifex the only source of translations  https://review.openstack.org/8419108:26
*** nprivalova has quit IRC08:29
*** ociuhandu has joined #openstack-infra08:31
*** _nadya_ has joined #openstack-infra08:37
*** sweston has quit IRC08:37
*** sweston has joined #openstack-infra08:38
*** afazekas has joined #openstack-infra08:42
*** jooools has joined #openstack-infra08:44
*** andreaf has quit IRC08:45
*** comstud has quit IRC08:48
*** comstud has joined #openstack-infra08:48
*** alexpilotti has joined #openstack-infra08:50
*** Ryan_Lane has joined #openstack-infra08:54
*** comstud has quit IRC08:57
*** mihgen has joined #openstack-infra09:00
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Minor fixes in the installation documentation  https://review.openstack.org/8588609:01
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency  https://review.openstack.org/8249209:01
*** Ryan_Lane has quit IRC09:03
*** comstud has joined #openstack-infra09:08
openstackgerritA change was merged to openstack-infra/config: Add jobs to publish nodepool docs  https://review.openstack.org/8485709:08
*** pcrews_ has joined #openstack-infra09:09
*** pcrews has quit IRC09:12
openstackgerritA change was merged to openstack-infra/config: Disable pypy jobs in ironic-python-agent  https://review.openstack.org/8578709:12
*** e0ne has joined #openstack-infra09:13
*** andreaf has joined #openstack-infra09:13
*** denis_makogon has quit IRC09:13
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Fix some minor typos in the trigger documentation  https://review.openstack.org/8588509:14
*** pcrews_ has quit IRC09:19
*** sweston has quit IRC09:21
*** pcrews_ has joined #openstack-infra09:21
*** ihrachyshka has quit IRC09:26
*** comstud has quit IRC09:28
rcarrillocruzclarkb , jeblair, fungi, mordred : hi guys, any of you still around?09:28
*** comstud has joined #openstack-infra09:28
rcarrillocruzi ran the review.pp manifest in a VM and I notice that it created home folders for your users, is this expected? Otherwise I'd open a bug and look at the manifests to see where those user home folders are created....09:29
*** e0ne_ has joined #openstack-infra09:30
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: Sort ant builders settings for consistency  https://review.openstack.org/8249209:30
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Do not use obsolete translations  https://review.openstack.org/8597309:32
*** e0ne has quit IRC09:33
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Do not generate obsolete translations  https://review.openstack.org/8597309:33
*** sweston has joined #openstack-infra09:34
*** comstud has quit IRC09:39
*** saju_m has quit IRC09:39
*** saschpe has quit IRC09:39
*** matsuhashi has quit IRC09:39
*** ttx has quit IRC09:39
*** dolphm has quit IRC09:39
*** Shrews has quit IRC09:39
*** mattoliverau has quit IRC09:39
*** yamahata has quit IRC09:39
*** jhesketh has quit IRC09:39
*** mgagne has quit IRC09:39
*** harlowja_away has quit IRC09:39
*** russellb has quit IRC09:39
*** sahumada has quit IRC09:39
*** zul has quit IRC09:39
*** bogdando has quit IRC09:39
*** annegentle_ has quit IRC09:39
*** markmc has quit IRC09:39
*** adam_g has quit IRC09:39
*** stevebaker has quit IRC09:39
*** jbryce has quit IRC09:39
*** thingee has quit IRC09:39
*** wendar_ has quit IRC09:39
*** Guest21911 has quit IRC09:39
*** pfallenop has quit IRC09:39
*** nibalizer has quit IRC09:39
*** dstanek has quit IRC09:39
*** russell_h has quit IRC09:39
*** jlk has quit IRC09:39
*** rainya- has quit IRC09:39
*** hughsaunders has quit IRC09:39
*** zz_gondoi has quit IRC09:39
*** Steap has quit IRC09:39
*** gaelL has quit IRC09:39
*** dguerri_ has quit IRC09:39
*** dtroyer has quit IRC09:39
*** alaski has quit IRC09:39
*** gmoro has quit IRC09:39
*** jroll has quit IRC09:39
*** EmilienM has quit IRC09:39
*** pvo has quit IRC09:39
*** uvirtbot has quit IRC09:39
*** westmaas has quit IRC09:39
*** tteggel has quit IRC09:39
*** dosaboy has quit IRC09:39
*** dhellmann has quit IRC09:39
*** flaper87|afk has quit IRC09:39
*** nijaba has quit IRC09:39
*** odyi has quit IRC09:39
*** tristanC has quit IRC09:39
*** dangers_away has quit IRC09:39
*** milki has quit IRC09:39
*** maurosr has quit IRC09:39
*** dragondm has quit IRC09:39
*** stevebaker has joined #openstack-infra09:39
*** adam_g has joined #openstack-infra09:39
*** adam_g has quit IRC09:39
*** adam_g has joined #openstack-infra09:39
*** pfallenop has joined #openstack-infra09:39
*** milki has joined #openstack-infra09:39
*** markmc has joined #openstack-infra09:39
*** odyi has joined #openstack-infra09:39
*** flaper87|afk has joined #openstack-infra09:39
*** Steap has joined #openstack-infra09:39
*** bogdando has joined #openstack-infra09:39
*** gmoro has joined #openstack-infra09:39
*** matsuhashi has joined #openstack-infra09:39
*** mattoliverau has joined #openstack-infra09:39
*** EmilienM has joined #openstack-infra09:39
*** tteggel has joined #openstack-infra09:39
*** saju_m has joined #openstack-infra09:39
*** odyi has quit IRC09:39
*** bogdando has quit IRC09:39
*** sweston has quit IRC09:39
*** pcrews_ has quit IRC09:39
*** jcoufal has quit IRC09:39
*** yfried has quit IRC09:39
*** clarkb has quit IRC09:39
*** david-lyle has quit IRC09:39
*** openstackgerrit has quit IRC09:39
*** SumitNaiksatam has quit IRC09:39
*** jamespage has quit IRC09:39
*** zhiyan has quit IRC09:39
*** moted has quit IRC09:39
*** sdake has quit IRC09:39
*** thomasem has quit IRC09:39
*** rfolco has quit IRC09:39
*** miarmak has quit IRC09:39
*** fifieldt has quit IRC09:39
*** kevinbenton has quit IRC09:39
*** slagle has quit IRC09:39
*** flaper87|afk is now known as flaper8709:39
*** miarmak has joined #openstack-infra09:39
*** zul has joined #openstack-infra09:39
*** openstack has joined #openstack-infra09:50
*** unicell has joined #openstack-infra09:50
*** kiall has joined #openstack-infra09:51
*** soren has joined #openstack-infra09:51
*** che-arne has joined #openstack-infra09:52
*** wayneeseguin has joined #openstack-infra09:52
*** matsuhashi has quit IRC09:52
*** mikal has joined #openstack-infra09:52
*** bnemec has joined #openstack-infra09:52
*** flaper87 has quit IRC09:53
*** flaper87 has joined #openstack-infra09:53
*** slagle has joined #openstack-infra09:53
*** koolhead17 has joined #openstack-infra09:53
*** matsuhashi has joined #openstack-infra09:53
*** jp_at_hp has quit IRC09:55
*** jp_at_hp has joined #openstack-infra09:55
*** adarazs has quit IRC09:56
*** adarazs has joined #openstack-infra09:56
*** jeckersb has quit IRC09:56
*** jeckersb has joined #openstack-infra09:56
*** yfried has quit IRC09:56
*** yfried has joined #openstack-infra09:56
*** SumitNaiksatam has quit IRC09:56
*** SumitNaiksatam has joined #openstack-infra09:56
*** jcoufal has quit IRC09:56
*** jcoufal has joined #openstack-infra09:56
*** dangers_away has quit IRC09:56
*** dangers_away has joined #openstack-infra09:56
*** kmartin has quit IRC09:57
*** kmartin has joined #openstack-infra09:57
*** tteggel has quit IRC09:57
*** tteggel has joined #openstack-infra09:57
*** mkerrin has quit IRC09:57
*** mkerrin has joined #openstack-infra09:57
*** maurosr has quit IRC09:57
*** maurosr has joined #openstack-infra09:57
*** jlibosva has quit IRC09:57
*** jlibosva has joined #openstack-infra09:57
*** che-arne has quit IRC09:59
*** che-arne has joined #openstack-infra09:59
*** Ryan_Lane has joined #openstack-infra09:59
*** persia has quit IRC10:00
*** persia has joined #openstack-infra10:01
*** persia has quit IRC10:01
*** persia has joined #openstack-infra10:01
*** persia is now known as Guest1719810:01
*** Guest17198 has quit IRC10:01
*** Guest17198 has joined #openstack-infra10:01
*** Guest17198 is now known as persia10:02
*** persia has quit IRC10:03
*** wendar has quit IRC10:03
*** sileht has quit IRC10:03
*** greghaynes has quit IRC10:03
*** stevebaker has quit IRC10:03
*** fungi has quit IRC10:03
*** adam_g has quit IRC10:03
*** sc68cal has quit IRC10:03
*** mrda_away has quit IRC10:03
*** zigo has quit IRC10:03
*** StevenK has quit IRC10:03
*** jdurgin has quit IRC10:03
*** sdague has quit IRC10:03
*** bknudson has quit IRC10:03
*** yongli has quit IRC10:03
*** gothicmindfood has quit IRC10:03
*** ruhe has quit IRC10:03
*** dansmith has quit IRC10:03
*** JoshNang has quit IRC10:03
*** pbelanyi has quit IRC10:03
*** ihrachyshka has joined #openstack-infra10:03
*** adam_g` has joined #openstack-infra10:03
*** persia has joined #openstack-infra10:03
*** stevebaker has joined #openstack-infra10:03
*** dansmith has joined #openstack-infra10:03
*** StevenK has joined #openstack-infra10:03
*** mrda_away has joined #openstack-infra10:03
*** JoshNang has joined #openstack-infra10:03
*** yongli has joined #openstack-infra10:03
*** greghaynes has joined #openstack-infra10:03
*** persia has quit IRC10:03
*** persia has joined #openstack-infra10:03
*** zhiyan is now known as zhiyan_10:03
*** ruhe has joined #openstack-infra10:04
*** sdague has joined #openstack-infra10:04
*** fungi has joined #openstack-infra10:04
*** gothicmindfood has joined #openstack-infra10:04
*** sc68cal has joined #openstack-infra10:04
*** sileht has joined #openstack-infra10:04
*** wendar has joined #openstack-infra10:04
*** zigo has joined #openstack-infra10:04
*** Ryan_Lane has quit IRC10:04
*** jdurgin has joined #openstack-infra10:04
*** pbelanyi has joined #openstack-infra10:04
sdaguemikal: what's up with turbohipster being drunk?10:05
*** Ng has quit IRC10:08
*** jgallard has quit IRC10:09
*** Ng has joined #openstack-infra10:09
*** yassine has joined #openstack-infra10:09
*** pvo has quit IRC10:13
*** jbryce has quit IRC10:13
*** russell_h has quit IRC10:13
*** annegentle has quit IRC10:13
*** dolphm has quit IRC10:13
*** wendar has quit IRC10:14
*** wendar has joined #openstack-infra10:14
*** StevenK has quit IRC10:15
*** ttx has quit IRC10:15
*** russellb has quit IRC10:15
*** rainya has quit IRC10:15
*** jlk has quit IRC10:15
*** gondoi has quit IRC10:15
*** Guest99168 has quit IRC10:15
*** fungi has quit IRC10:15
*** stevebaker has quit IRC10:15
*** adam_g` has quit IRC10:15
*** JayF has quit IRC10:15
*** EmilienM has quit IRC10:15
*** jroll has quit IRC10:15
*** comstud has quit IRC10:15
*** alaski has quit IRC10:15
*** StevenK has joined #openstack-infra10:15
*** reed_ has quit IRC10:16
*** oomichi has joined #openstack-infra10:16
*** mgagne has quit IRC10:16
*** _nadya_ has quit IRC10:16
*** adam_g has joined #openstack-infra10:20
*** adam_g has quit IRC10:20
*** adam_g has joined #openstack-infra10:20
*** stevebaker has joined #openstack-infra10:20
*** ttx has joined #openstack-infra10:20
*** JayF has joined #openstack-infra10:21
*** JayF has quit IRC10:21
*** JayF has joined #openstack-infra10:21
*** EmilienM has joined #openstack-infra10:22
*** NikitaKonovalov has quit IRC10:26
*** DinaBelova has quit IRC10:26
*** sdake_ has quit IRC10:26
*** medieval1 has quit IRC10:26
*** jhesketh has quit IRC10:26
*** bogdando has quit IRC10:26
*** 6JTAANJSJ has quit IRC10:26
*** uvirtbot has quit IRC10:26
*** zhiyan_ has quit IRC10:26
*** jcoufal has quit IRC10:26
*** michchap has quit IRC10:26
*** Shrews has quit IRC10:26
*** kevinbenton has quit IRC10:26
*** gema has quit IRC10:26
*** fifieldt has quit IRC10:26
*** andreaf has quit IRC10:26
*** alexpilotti has quit IRC10:26
*** jooools has quit IRC10:26
*** jp_at_hp has quit IRC10:26
*** ominakov has quit IRC10:26
*** amotoki has quit IRC10:26
*** imcsk8_lap has quit IRC10:26
*** yaguang has quit IRC10:26
*** adalbas has quit IRC10:26
*** ogelbukh has quit IRC10:26
*** eloib_ has quit IRC10:26
*** pabelanger has quit IRC10:26
*** primeministerp has quit IRC10:26
*** kashyap has quit IRC10:26
*** jgriffith has quit IRC10:26
*** locke105 has quit IRC10:26
*** Adri2000 has quit IRC10:26
*** mtreinish has quit IRC10:26
*** katyafervent has quit IRC10:26
*** mburned has quit IRC10:26
*** hdd has quit IRC10:26
*** mika has quit IRC10:26
*** vladan has quit IRC10:26
*** Mithrandir has quit IRC10:26
*** spredzy has quit IRC10:26
*** rcarrillocruz has quit IRC10:26
*** simonmcc_ has quit IRC10:26
*** kevinbenton has joined #openstack-infra10:27
*** imcsk8_lap has joined #openstack-infra10:27
*** gema has joined #openstack-infra10:27
*** mika has joined #openstack-infra10:27
*** mika has joined #openstack-infra10:27
*** simonmcc_ has joined #openstack-infra10:27
*** zehicle has joined #openstack-infra10:27
*** pabelanger has joined #openstack-infra10:27
*** uvirtbot has joined #openstack-infra10:27
*** hdd has joined #openstack-infra10:27
*** ogelbukh has joined #openstack-infra10:27
*** adalbas has joined #openstack-infra10:27
*** katyafervent has joined #openstack-infra10:27
*** mtreinish has joined #openstack-infra10:27
*** bogdando has joined #openstack-infra10:27
*** amotoki has joined #openstack-infra10:27
*** jp_at_hp has joined #openstack-infra10:27
*** rcarrillocruz has joined #openstack-infra10:27
*** sdake_ has joined #openstack-infra10:27
*** spredzy has joined #openstack-infra10:27
*** simonmcc_ has quit IRC10:27
*** simonmcc_ has joined #openstack-infra10:27
*** NikitaKonovalov has joined #openstack-infra10:27
*** michchap has joined #openstack-infra10:27
*** ominakov has joined #openstack-infra10:27
*** jooools has joined #openstack-infra10:27
*** alexpilotti has joined #openstack-infra10:27
*** andreaf has joined #openstack-infra10:27
*** jgriffit1 has joined #openstack-infra10:27
*** Mithrandir has joined #openstack-infra10:27
*** jp_at_hp has quit IRC10:27
*** jp_at_hp has joined #openstack-infra10:27
*** vladan has joined #openstack-infra10:27
*** andreaf has quit IRC10:27
*** andreaf has joined #openstack-infra10:27
*** medieval1 has joined #openstack-infra10:27
*** hdd has quit IRC10:27
*** hdd has joined #openstack-infra10:27
*** jp_at_hp has quit IRC10:27
*** jp_at_hp has joined #openstack-infra10:27
*** jp_at_hp has quit IRC10:28
*** jp_at_hp has joined #openstack-infra10:28
*** Shrews has joined #openstack-infra10:28
*** zhiyan_ has joined #openstack-infra10:28
*** DinaBelova has joined #openstack-infra10:28
mikalsdague: turbo-hipster had some issues with the grizzly branch being deleted from git. They should be resolved now.10:28
*** kashyap has joined #openstack-infra10:28
*** yaguang has joined #openstack-infra10:28
sdagueok, it just failed me again on master10:28
*** jcoufal has joined #openstack-infra10:28
sdagueunless it doesn't do rechecks?10:29
*** jhesketh has joined #openstack-infra10:29
*** fifieldt has joined #openstack-infra10:29
sdagueI guess that's an old vote10:29
*** fungi has joined #openstack-infra10:29
*** eloib_ has joined #openstack-infra10:29
sdaguemikal: it worth retriggering t-h results on all the -1s? I know I tend to filter out -1s so all the changes from yesterday will be masked for a lot of people10:30
*** Adri2000 has joined #openstack-infra10:30
*** Adri2000 has quit IRC10:30
*** Adri2000 has joined #openstack-infra10:31
*** locke105 has joined #openstack-infra10:31
*** Adri2000 is now known as Guest4410610:31
mikalsdague: I believe jhesketh has already done that10:31
mikalsdague: recheck is definitely supported10:31
mikalGot a review URL I can take a look at?10:31
jheskethsdague: yep, should re re-running10:31
sdaguemikal: https://review.openstack.org/#/c/85780/10:32
mikalLOL, that guy10:32
mikalAhhh, failure was from before jhesketh fixed this. The recheck is still running.10:33
*** primeministerp has joined #openstack-infra10:34
*** saju_m has quit IRC10:34
*** saju_m has joined #openstack-infra10:35
*** mburned has joined #openstack-infra10:36
*** reed has joined #openstack-infra10:36
sdagueok, no prob10:36
mikalSorry for any inconvenience, and please come again!10:36
mikaljhesketh: we should file a bug to make sure this doesn't happen again10:36
*** Ryan_Lane has joined #openstack-infra10:36
*** rcarrillocruz has quit IRC10:39
*** zehicle has quit IRC10:39
*** sdake_ has quit IRC10:39
*** mika has quit IRC10:39
*** gothicmindfood has quit IRC10:39
*** slagle has quit IRC10:39
*** soren has quit IRC10:39
*** bookwar has quit IRC10:39
*** maurosr has quit IRC10:39
*** vponomaryov has quit IRC10:39
*** tteggel has quit IRC10:39
*** aglarendil has quit IRC10:39
*** sdake has quit IRC10:39
*** jamespage has quit IRC10:39
*** antonym has quit IRC10:39
*** david-lyle has quit IRC10:39
*** yfried has quit IRC10:39
*** jeckersb has quit IRC10:39
*** miarmak has quit IRC10:39
*** mattoliverau has quit IRC10:39
*** flaper87 has quit IRC10:39
*** pfallenop has quit IRC10:39
*** ociuhandu has quit IRC10:39
*** hashar has quit IRC10:39
*** dcramer_ has quit IRC10:39
*** msabramo has quit IRC10:39
*** gnuoy has quit IRC10:39
*** MIDENN_ has quit IRC10:39
*** Guest456 has quit IRC10:39
*** GheRivero has quit IRC10:39
*** mwagner_lap has quit IRC10:39
*** asettle has quit IRC10:39
*** jamespage has joined #openstack-infra10:39
*** mika has joined #openstack-infra10:39
*** mika has quit IRC10:39
*** mika has joined #openstack-infra10:39
*** msabramo has joined #openstack-infra10:39
*** jamespage has quit IRC10:39
*** jamespage has joined #openstack-infra10:39
*** slagle has joined #openstack-infra10:39
*** aglarendil has joined #openstack-infra10:39
*** mattoliverau has joined #openstack-infra10:40
*** GheRivero has joined #openstack-infra10:40
*** jeckersb has joined #openstack-infra10:40
*** miarmak has joined #openstack-infra10:40
*** dcramer_ has joined #openstack-infra10:40
*** zehicle has joined #openstack-infra10:40
*** yfried has joined #openstack-infra10:40
*** MIDENN_ has joined #openstack-infra10:40
*** gnuoy has joined #openstack-infra10:40
*** rcarrillocruz has joined #openstack-infra10:40
*** gothicmindfood has joined #openstack-infra10:40
*** soren has joined #openstack-infra10:40
*** asettle has joined #openstack-infra10:40
*** mwagner_lap has joined #openstack-infra10:40
*** sdake_ has joined #openstack-infra10:40
*** sdake has joined #openstack-infra10:40
*** Vivek has joined #openstack-infra10:40
*** antonym has joined #openstack-infra10:40
*** vponomaryov has joined #openstack-infra10:40
*** hashar has joined #openstack-infra10:40
*** jeckersb has quit IRC10:40
*** jeckersb has joined #openstack-infra10:40
*** david-lyle has joined #openstack-infra10:40
*** ociuhandu has joined #openstack-infra10:40
*** dcramer_ has quit IRC10:40
*** dcramer_ has joined #openstack-infra10:40
*** pfallenop has joined #openstack-infra10:40
*** MIDENN_ has quit IRC10:40
*** MIDENN_ has joined #openstack-infra10:40
*** flaper87 has joined #openstack-infra10:40
*** yfried has quit IRC10:40
*** yfried has joined #openstack-infra10:40
*** Vivek is now known as Guest2677110:40
*** bookwar has joined #openstack-infra10:40
*** tteggel has joined #openstack-infra10:40
*** maurosr has joined #openstack-infra10:40
*** flaper87 has quit IRC10:40
*** flaper87 has joined #openstack-infra10:40
*** maurosr has quit IRC10:40
*** maurosr has joined #openstack-infra10:40
*** flaper87 has quit IRC10:40
*** flaper87 has joined #openstack-infra10:40
*** tteggel has quit IRC10:41
*** tteggel has joined #openstack-infra10:41
*** Ryan_Lane has quit IRC10:41
jheskethsdague: there you go, turbo-hipster +110:42
jheskethmikal: do you mean a bug for why it failed?10:42
mikaljhesketh: yeah, a bug to remind us to try for tags as well as branches10:42
*** AJaeger has joined #openstack-infra10:44
* mikal goes back to his movie10:44
*** matsuhashi has quit IRC10:53
*** matsuhashi has joined #openstack-infra10:54
*** sdake has quit IRC10:57
*** sandywalsh has joined #openstack-infra10:58
*** matsuhashi has quit IRC10:58
*** matsuhashi has joined #openstack-infra10:59
*** ildikov_ has joined #openstack-infra11:02
*** chandan_kumar has joined #openstack-infra11:03
*** hashar has quit IRC11:14
*** julienvey has joined #openstack-infra11:21
*** e0ne_ has quit IRC11:22
*** e0ne has joined #openstack-infra11:23
*** e0ne has quit IRC11:27
*** sdake has joined #openstack-infra11:31
*** yaguang has quit IRC11:32
*** Ryan_Lane has joined #openstack-infra11:37
*** weshay has joined #openstack-infra11:41
*** Ryan_Lane has quit IRC11:42
*** chandan_kumar has quit IRC11:44
*** alaski has joined #openstack-infra11:44
*** jroll has joined #openstack-infra11:45
*** zz_gondoi has joined #openstack-infra11:46
*** dolphm has joined #openstack-infra11:46
*** annegentle has joined #openstack-infra11:46
*** zz_gondoi is now known as gondoi11:46
*** russellb has joined #openstack-infra11:46
*** jbryce has joined #openstack-infra11:46
*** russell_h has joined #openstack-infra11:46
*** salv-orlando has joined #openstack-infra11:47
*** mgagne has joined #openstack-infra11:48
*** lillie has joined #openstack-infra11:48
*** pvo has joined #openstack-infra11:48
*** lillie is now known as Guest1926911:49
*** jlk has joined #openstack-infra11:52
*** jlk has joined #openstack-infra11:52
*** fifieldt has quit IRC11:52
*** comstud has joined #openstack-infra11:52
*** e0ne has joined #openstack-infra11:53
*** jgallard has joined #openstack-infra11:55
*** e0ne has quit IRC11:58
*** pdmars has joined #openstack-infra11:59
*** mbacchi has joined #openstack-infra12:00
*** markmcclain has joined #openstack-infra12:00
*** chandan_kumar has joined #openstack-infra12:02
*** ttx has quit IRC12:03
*** ttx has joined #openstack-infra12:03
*** salv-orlando has quit IRC12:07
*** pcrews__ has joined #openstack-infra12:07
*** sdake has quit IRC12:09
*** pcrews_ has quit IRC12:10
*** che-arne has quit IRC12:13
*** pcm has joined #openstack-infra12:14
*** hdd has quit IRC12:14
*** simonmcc_ has quit IRC12:16
*** gondoi has quit IRC12:17
*** sc68cal has quit IRC12:17
*** wendar has quit IRC12:17
*** david-lyle has quit IRC12:17
*** jbryce has quit IRC12:17
*** annegentle has quit IRC12:17
*** simonmcc_ has joined #openstack-infra12:17
*** zigo has quit IRC12:17
*** persia has quit IRC12:17
*** Guest19269 has quit IRC12:17
*** pvo has quit IRC12:17
*** russell_h has quit IRC12:17
*** pabelanger has quit IRC12:17
*** wendar has joined #openstack-infra12:17
*** persia has joined #openstack-infra12:18
*** matsuhashi has quit IRC12:18
*** persia has quit IRC12:18
*** persia has joined #openstack-infra12:18
*** zigo has joined #openstack-infra12:18
*** comstud has quit IRC12:18
*** dolphm has quit IRC12:18
*** AJaeger has quit IRC12:18
*** jlk has quit IRC12:18
*** russellb has quit IRC12:18
*** matsuhashi has joined #openstack-infra12:18
*** adam_g has quit IRC12:18
*** dkranz has joined #openstack-infra12:19
*** salv-orlando has joined #openstack-infra12:19
*** jroll has quit IRC12:19
*** alaski has quit IRC12:19
*** stevebaker has quit IRC12:19
*** StevenK has quit IRC12:19
*** sileht has quit IRC12:19
*** greghaynes has quit IRC12:19
*** mgagne has quit IRC12:19
*** JayF has quit IRC12:19
*** jdurgin has quit IRC12:19
*** matsuhashi has quit IRC12:19
*** matsuhashi has joined #openstack-infra12:19
*** StevenK has joined #openstack-infra12:20
*** JayF has joined #openstack-infra12:20
*** stevebaker has joined #openstack-infra12:20
*** comstud has joined #openstack-infra12:20
*** alaski has joined #openstack-infra12:20
*** jlk has joined #openstack-infra12:20
*** mkoderer has joined #openstack-infra12:20
*** jlk has quit IRC12:20
*** jlk has joined #openstack-infra12:20
*** AJaeger has joined #openstack-infra12:21
*** AJaeger has joined #openstack-infra12:21
*** mgagne has joined #openstack-infra12:21
*** greghaynes has joined #openstack-infra12:21
*** dolphm has joined #openstack-infra12:21
*** adam_g has joined #openstack-infra12:21
*** adam_g has quit IRC12:21
*** adam_g has joined #openstack-infra12:21
*** sc68cal has joined #openstack-infra12:21
*** jroll has joined #openstack-infra12:21
*** russell_h has joined #openstack-infra12:21
*** jdurgin has joined #openstack-infra12:21
*** annegentle has joined #openstack-infra12:21
*** russellb has joined #openstack-infra12:21
*** jbryce has joined #openstack-infra12:21
*** sileht has joined #openstack-infra12:22
*** lillie- has joined #openstack-infra12:22
*** pabelanger has joined #openstack-infra12:22
*** pvo has joined #openstack-infra12:23
*** e0ne has joined #openstack-infra12:23
*** gondoi has joined #openstack-infra12:23
*** freyes has quit IRC12:23
*** _nadya_ has joined #openstack-infra12:23
*** freyes has joined #openstack-infra12:26
*** yfried_ has joined #openstack-infra12:26
*** zhiyan_ is now known as zhiyan12:27
*** ihrachyshka has quit IRC12:27
*** yfried has quit IRC12:27
*** e0ne has quit IRC12:28
*** dcramer_ has quit IRC12:30
*** matsuhashi has quit IRC12:30
*** aysyd has joined #openstack-infra12:31
*** salv-orlando has quit IRC12:32
*** changbl has quit IRC12:32
pcmsdague: ping12:34
*** rlandy has joined #openstack-infra12:35
*** bknudson has joined #openstack-infra12:37
*** eharney has joined #openstack-infra12:37
*** Ryan_Lane has joined #openstack-infra12:38
*** Ryan_Lane has joined #openstack-infra12:38
*** miqui has joined #openstack-infra12:38
*** jroll has quit IRC12:38
*** jroll has joined #openstack-infra12:39
NobodyCamgood morning Ironic12:39
*** e0ne has joined #openstack-infra12:39
*** blamar has joined #openstack-infra12:40
sdaguepcm: pong12:40
*** Ryan_Lane has quit IRC12:42
*** freyes has quit IRC12:43
*** heyongli has joined #openstack-infra12:44
*** yamahata has joined #openstack-infra12:46
*** yfried__ has joined #openstack-infra12:48
*** maxbit has joined #openstack-infra12:48
*** ameade has joined #openstack-infra12:49
*** lillie- is now known as lillie12:49
*** lillie is now known as Guest3743212:50
NobodyCamdoh .... good morning infra :-p12:51
*** thuc has joined #openstack-infra12:51
*** yfried_ has quit IRC12:52
*** thuc has quit IRC12:52
*** thuc_ has joined #openstack-infra12:52
*** thuc has joined #openstack-infra12:54
*** dkranz has quit IRC12:55
*** dstanek has quit IRC12:55
*** dstanek has joined #openstack-infra12:56
*** pblaho has joined #openstack-infra12:57
SergeyLukjanovNobodyCam, morning ;)12:57
*** thuc_ has quit IRC12:57
*** fifieldt has joined #openstack-infra12:57
NobodyCammorning SergeyLukjanov :)12:57
*** maxbit has quit IRC12:58
pcmNow that Juno is open, I'd like to bring up https://bugs.launchpad.net/neutron/+bug/1282855 for consideration of httmock in test-requirements.12:58
uvirtbotLaunchpad bug 1282855 in neutron "Add httmock to test-requirements.txt and update requests -> 2.1.0" [Undecided,In progress]12:58
pcmsdague: ^^12:59
*** _nadya_ has quit IRC13:00
*** signed8bit has joined #openstack-infra13:01
sdaguepcm: could you start that with an ML thread on the list. I feel we've got a lot of different mocking libraries, and we should figure out if we can converge13:01
sdaguebecause if test code is different in every project that inhibits contributors13:01
SergeyLukjanovfungi, clarkb, jeblair, mordred, gerrit bot is dead...13:01
pcmI did send out an e-mail on the ML a few days ago.13:01
pcm4/4/2014 actually.13:02
*** dprince has joined #openstack-infra13:02
pcmsdague: http://lists.openstack.org/pipermail/openstack-dev/2014-April/031901.html13:05
sdagueok but it's a neutron only tag13:06
*** jaypipes has joined #openstack-infra13:06
pcmon the ML? I have [infra][neutron]. Or do you mean the bug?13:06
sdaguethe ML post13:06
*** timrc-afk is now known as timrc13:06
pcmNot sure how to do this. Can you please advise?13:07
pcm(as in not sure how to get a proposal for a new package presented - been floundering for several months now)13:08
sdaguepcm: well how about figuring out who's using httpretty at this point, because the 2 seem to largely overlap in function, and we're trying to not just make the requirements explode13:11
*** andreykurilin_ has joined #openstack-infra13:14
*** _nadya_ has joined #openstack-infra13:14
pcmsdague: ok. Once found, what is the next step?13:15
*** miqui has quit IRC13:15
sdaguepcm: get an agreement to move to this instead13:15
*** miqui has joined #openstack-infra13:16
sdagueif you are proposing a new requirement that's largely duplicative of an existing requirement, the burden has to be on the proposer to get the duplicative requirement deprecated across openstack13:16
sdagueotherwise requirements grow without bound as everyone brings in their favorite libraries, and the projects diverge13:17
pcmok. Thanks for the info.13:17
*** unicell has quit IRC13:17
*** unicell has joined #openstack-infra13:18
*** unicell has quit IRC13:19
*** lcostantino has joined #openstack-infra13:19
dhellmannSergeyLukjanov: freenode restarted a bunch of servers overnight because of the OpenSSL bug13:19
*** unicell has joined #openstack-infra13:19
pcmWill either spend the effort to push this, or give up and try to make httpretty work.13:19
SergeyLukjanovdhellmann, it was working about 4 hours ago13:20
*** unicell has quit IRC13:20
dhellmannSergeyLukjanov: ok, I saw a message about restarts on twitter but I don't know what time that was sent13:20
SergeyLukjanovdhellmann, anyway, someone should kick gerrit bot13:21
SergeyLukjanovI mean foot kick :)13:21
fungiokay, what's splodey today?13:21
dhellmannSergeyLukjanov: heh13:21
fungiany fallout from all our updates last night?13:21
*** pcargnel has joined #openstack-infra13:21
SergeyLukjanovfungi, morning13:21
SergeyLukjanovfungi, everything looks ok13:22
*** homeless has joined #openstack-infra13:22
fungiSergeyLukjanov: thanks--that's excellent news. maybe i'll get some work done then ;)13:22
SergeyLukjanovfungi, gerrit bot is slacking again13:22
fungioh, just saw that--fixing now13:22
*** signed8bit has quit IRC13:22
SergeyLukjanovfungi, thx13:23
*** signed8bit has joined #openstack-infra13:23
*** pcargnel has quit IRC13:23
SergeyLukjanovfungi, probably we should make it fail-fast and run under the "forever"13:24
*** openstackgerrit has joined #openstack-infra13:24
fungiopenstackgerrit has returned13:24
fungiopenstackstatus (statusbot) and openstack (meetbot) are still here, so i think all is well in ircdom13:25
*** mfer has joined #openstack-infra13:26
*** fbo is now known as fbo_away13:27
SergeyLukjanovsdague, are you ok with https://review.openstack.org/#/c/84840/?13:28
*** _nadya_ has quit IRC13:28
*** dcramer_ has joined #openstack-infra13:29
*** pcm has left #openstack-infra13:30
sdagueSergeyLukjanov: I think there are more things that should be there13:30
*** skraynev is now known as skraynev_afk13:31
*** dizquierdo is now known as dizquierdo_afk13:32
*** dims has joined #openstack-infra13:33
*** mrmartin has joined #openstack-infra13:34
sdaguefungi: man, gerrit is so slow again13:34
sdaguedid we every figure out if there was a way to instrument that?13:35
fungislow to respond to api calls?13:35
*** miqui_ has joined #openstack-infra13:36
*** miqui has quit IRC13:37
*** Ryan_Lane has joined #openstack-infra13:39
*** julim has joined #openstack-infra13:39
anteayamattoliverau: looking for your email regarding your ballot, I don't see anything. I am concluding you have sorted out the answer your needed on your own.13:39
*** thomasem has joined #openstack-infra13:41
*** nkinder has quit IRC13:41
sdaguefungi: git review takes > 30s13:41
sdagueto push a 10 line patch13:41
*** oomichi has quit IRC13:42
*** jcoufal has quit IRC13:42
*** zehicle has quit IRC13:43
*** zehicle_at_dell has quit IRC13:43
*** thuc_ has joined #openstack-infra13:43
*** Ryan_Lane has quit IRC13:43
fungithe jvm for gerrit is running system load up to ~313:43
fungithough the server has 8vcpus, so that should be plenty low13:44
*** jcoufal has joined #openstack-infra13:44
sdaguefungi: ok, is that something which is monitored in cacti? hard to kmow if that's good or bad13:44
*** ihrachyshka has joined #openstack-infra13:44
sdaguewhat all did you end up having to revoke for infra based on the heartbug?13:44
*** dkranz has joined #openstack-infra13:46
*** thuc has quit IRC13:46
*** alaski has quit IRC13:46
*** alaski has joined #openstack-infra13:47
*** thuc_ has quit IRC13:47
*** hashar has joined #openstack-infra13:47
*** andreykurilin_ has quit IRC13:47
fungiit is monitored by cacti, but i can't look at the graphs for it right now because http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-2708.html13:48
uvirtbotfungi: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-2708)13:48
fungilast night we got through rekeying the puppet relationships for all machines after upgrading packages everywhere and rebooting for good measure. still plenty of things to revoke/regenerate/reissue/replace though13:49
fungibut the big scare was that we had an openssl-based socket listening on the puppetmaster, so keys to the kingdom were top priority13:50
*** alexpilotti has quit IRC13:51
fungithe other stuff we can update piecemeal with minimal impact to the community (still a priority, but doesn't need to happen in one shot or during maintenance windows)13:51
*** freyes_ has joined #openstack-infra13:52
sdaguefungi: gotcha. I'm trying to sort out if openvpn is actually impacted by this13:52
*** vhoward- has left #openstack-infra13:52
fungimainly want to regenerate ssh host keys, get x.509 certs for services reissued based on fresh keys, reset passwords for everything, and so on13:52
fungisdague: funny, i got asked the same thing last night by a friend who uses it to virtually network devices for his customers (manages smart climate control systems in residences and office/industrial buildings)13:54
fungifrom what i know of openvpn it uses openssl for its crypto primitives but doesn't *actually* do ssl/tls with it13:54
sdagueyeh, well my exposure changes if openvpn is or is not affected, like puppet. As my puppet happens only over openvpn links13:54
sdaguebecause it was simpler to just build a shared L2 that way, and not punch extra holes in my firewall13:55
fungialso, i suspect if you follow their recommendation to use a tls key like they recommend, it never actually gets to the handshake because openvpn discards datagrams which don't have the right secret on them13:56
*** pcrews__ has quit IRC13:56
fungilooking into it myself out of curiosity now, even though i don't really use openvpn for anything currently13:56
fungithough i used to manage large customer vpn aggregators based on it, so have a pretty strong understanding of its inner workings13:57
*** che-arne has joined #openstack-infra13:57
*** markmcclain has quit IRC13:58
*** prad has joined #openstack-infra13:59
*** zns has joined #openstack-infra14:00
*** hashar has quit IRC14:00
fungilots of news articles out there which mention that openvpn uses openssl, but no statement from the openvpn devs on actual viability of the exploit against it and if so what configuration options expose the behavior14:00
*** zns has quit IRC14:01
*** zns has joined #openstack-infra14:01
openstackgerritEric Windisch proposed a change to openstack-infra/config: Make docker py26/py27 gates voting  https://review.openstack.org/8604214:02
*** msabramo has quit IRC14:02
*** jgriffit1 has quit IRC14:03
*** jgriffit1 has joined #openstack-infra14:05
fungisdague: as i suspected, "Using the tls-auth option should protect against this vulnerability14:06
fungi(assuming that your tls-auth key is not known to the attacker)."14:06
*** malini_afk is now known as malini14:07
*** jergerber has joined #openstack-infra14:08
*** pdmars has quit IRC14:08
*** pdmars has joined #openstack-infra14:08
sdaguefungi: ok, good to know, however I didn't set that14:08
fungiyeah, i always used those and baked them into the auto-installed client config for good measure14:10
*** miqui_ is now known as miqui14:10
fungisince the howtos on configuring openvpn (back to the early days) recommended it as a belt-and-braces measure against ssl-layer vulnerabilities14:10
fungithough worth noting, it's a shared key amongst all clients to that endpoint (sort of similar to an ipsec xauth group key but at a lower level and of course entirely different protocol) so if you have a malicious client with a copy of that key you're still vulnerable to it at that point14:13
fungimerely reduces your exposure to those who know that key14:13
sdagueso you guys did host key resets on everything as well?14:15
fungithat's still in progress14:15
*** chandan_kumar has quit IRC14:16
*** rcleere has joined #openstack-infra14:16
fungii'm waffling on regenerating the api host key for review.openstack.org since i suspect that will be disruptive to the entire development community with long-lived repercussions14:16
sdaguewell thinking this through, isn't that the only vulnerable system14:17
sdaguebecause you'd need a valid connection for the heartbeat14:18
sdagueor am I thinking about that wrong14:18
fungino, the way this vulnerability works is by exploiting an uninitialized struct in the response packet during the handshake14:19
fungireminiscent of the old days when you could read system memory remotely by sending large icmp echo packets and looking at the padding in the echo replies14:19
*** heyongli has quit IRC14:20
fungithe response packet is related to feature option negotiation for tls heartbeat support, but doesn't actually use the heartbeat traffic14:20
*** che-arne has quit IRC14:20
fungialso, the ssh api wouldn't be directly vulnerable to this because it's not tls, nor is it probably even using openssl inside the gerrit jvm (though i suppose it could be at some level, not in a vulnerable manner)14:21
*** jgrimm has joined #openstack-infra14:22
fungithe reason to regenerate the host key for that service is purely because it's cached in memory on the host and thus could have been read out by exploiting the vulnerable https service running there before we upgraded it14:22
*** thuc has joined #openstack-infra14:24
*** freyes_ is now known as freyes14:24
fungior, in an even more convoluted scenario, someone reading key material off the puppetmaster could have somehow coerced themselves access to review.o.o at a level where they were able to retrieve its host key from the filesystem or something14:25
*** dkliban has joined #openstack-infra14:26
sdaguebut the exposure is only within the process memory right?14:26
sdaguebecause the vmm protects you from reading actual system memory, as that's a segmentation fault14:27
*** pblaho has quit IRC14:27
*** david-lyle has joined #openstack-infra14:28
*** thuc has quit IRC14:29
*** nkinder has joined #openstack-infra14:30
fungihopefully yes, though you're trusting malloc'd segments to be zeroed (which is expected, but depends on platform)14:31
*** msabramo has joined #openstack-infra14:33
fungion modern linux i expect it's probably safe, which is why the top priority concerns we have are secret data which could be used by/pass through the process linking openssl and doing tls socket handling14:34
dhellmannfungi: you mentioned regnerating ssh host keys, do we need to worry about individual user keys, too?14:36
*** e0ne has quit IRC14:39
fungidhellmann: i wouldn't unless you were keeping them somewhere accessible to a vulnerable service (a web-based password safe for example)14:39
*** e0ne has joined #openstack-infra14:39
*** Ryan_Lane has joined #openstack-infra14:39
fungirsa authentication is good about you not having to disclose your secret key to services authenticating you14:40
dhellmannfungi: thanks14:40
* dhellmann trusts that the fungi answering questions in this room right now is the one he knows, and the answers can therefore be trusted #security14:41
*** saju_m has quit IRC14:41
sdagueit's turtles all the way down14:41
dhellmannsdague: indeed14:42
*** salv-orlando has joined #openstack-infra14:42
*** e0ne has quit IRC14:43
*** sandywalsh_ has joined #openstack-infra14:43
*** Ryan_Lane has quit IRC14:44
*** thingee has quit IRC14:45
*** jlibosva has quit IRC14:45
*** mkerrin has quit IRC14:45
*** dhellmann has quit IRC14:45
*** juice has quit IRC14:45
*** jd__ has quit IRC14:45
*** gilliard has quit IRC14:45
*** dizquierdo_afk has quit IRC14:45
*** dangers_away has quit IRC14:45
*** akscram has quit IRC14:45
*** plomakin_ has quit IRC14:45
*** sirushti has quit IRC14:45
*** sahumada has quit IRC14:45
*** nibalizer has quit IRC14:45
*** gaelL has quit IRC14:45
*** dosaboy has quit IRC14:45
*** gmoro has quit IRC14:45
*** mihgen has quit IRC14:45
*** afazekas has quit IRC14:45
*** yolanda has quit IRC14:45
*** ianw has quit IRC14:45
*** jeblair has quit IRC14:45
*** marktraceur has quit IRC14:45
*** marun has quit IRC14:45
*** krotscheck has quit IRC14:45
*** ryanpetrello has quit IRC14:45
*** YorikSar has quit IRC14:45
*** mordred has quit IRC14:45
*** dteselki- has quit IRC14:45
*** jamielennox|away has quit IRC14:45
*** BobBallA1ay has quit IRC14:45
*** proffalken has quit IRC14:45
*** dachary has quit IRC14:45
*** timrc has quit IRC14:45
*** jeremyb has quit IRC14:45
*** skraynev_afk has quit IRC14:45
*** SergeyLukjanov has quit IRC14:45
*** SpamapS has quit IRC14:45
*** spiffxp has quit IRC14:45
*** jesusaurus has quit IRC14:45
*** guitarzan has quit IRC14:45
*** blkperl has quit IRC14:45
*** vipul has quit IRC14:45
*** morgabra has quit IRC14:45
*** wenlock has joined #openstack-infra14:46
*** dangers_away has joined #openstack-infra14:48
*** guitarzan has joined #openstack-infra14:48
*** gmoro has joined #openstack-infra14:48
*** blkperl has joined #openstack-infra14:49
*** proffalken has joined #openstack-infra14:49
*** dosaboy has joined #openstack-infra14:49
*** sirushti has joined #openstack-infra14:49
*** krotscheck has joined #openstack-infra14:49
*** skraynev_afk has joined #openstack-infra14:49
*** wenlock has quit IRC14:49
*** wenlock has joined #openstack-infra14:49
*** jeremyb has joined #openstack-infra14:49
*** YorikSar has joined #openstack-infra14:49
*** jamielennox|away has joined #openstack-infra14:49
*** dangers_away has quit IRC14:49
*** dangers_away has joined #openstack-infra14:49
*** akscram has joined #openstack-infra14:49
*** yolanda has joined #openstack-infra14:49
*** jamielennox|away has quit IRC14:49
*** jamielennox|away has joined #openstack-infra14:49
*** dizquierdo has joined #openstack-infra14:49
*** ryanpetrello has joined #openstack-infra14:49
*** SergeyLukjanov has joined #openstack-infra14:49
*** spiffxp has joined #openstack-infra14:49
*** afazekas has joined #openstack-infra14:49
*** jlibosva has joined #openstack-infra14:49
*** timrc has joined #openstack-infra14:49
*** HenryG has quit IRC14:49
*** sahumada has joined #openstack-infra14:49
*** sandywalsh_ has quit IRC14:49
*** dhellmann has joined #openstack-infra14:49
*** vipul has joined #openstack-infra14:49
*** plomakin has joined #openstack-infra14:49
*** BobBallAway has joined #openstack-infra14:49
*** dachary has joined #openstack-infra14:50
*** jd__ has joined #openstack-infra14:50
*** morgabra has joined #openstack-infra14:50
*** dteselkin_ has joined #openstack-infra14:50
*** juice has joined #openstack-infra14:50
*** jesusaurus has joined #openstack-infra14:50
*** skolekonov has joined #openstack-infra14:50
*** SpamapS has joined #openstack-infra14:50
*** jeblair has joined #openstack-infra14:50
*** marktraceur has joined #openstack-infra14:50
*** ianw has joined #openstack-infra14:51
*** thedodd has joined #openstack-infra14:51
*** mordred has joined #openstack-infra14:51
*** gaelL has joined #openstack-infra14:52
*** nibalizer has joined #openstack-infra14:53
*** esker has joined #openstack-infra14:54
*** thingee has joined #openstack-infra14:55
*** mkerrin has joined #openstack-infra14:55
*** skolekonov has quit IRC14:58
*** thuc has joined #openstack-infra14:59
*** HenryG has joined #openstack-infra14:59
*** e0ne has joined #openstack-infra14:59
*** thuc_ has joined #openstack-infra15:00
*** wchrisj has joined #openstack-infra15:00
*** sabari2 has joined #openstack-infra15:02
*** gilliard has joined #openstack-infra15:02
*** gokrokve has joined #openstack-infra15:03
*** e0ne_ has joined #openstack-infra15:03
*** russell_h has quit IRC15:03
*** russell_h has joined #openstack-infra15:03
*** thuc has quit IRC15:03
*** hashar has joined #openstack-infra15:05
*** pcrews has joined #openstack-infra15:05
*** e0ne has quit IRC15:06
mordredfungi: I would not expect malloc'd segments to be zeroed15:06
mordredfungi: one of the big differences between new in C++ and malloc in C is that new _Does_ zero and malloc does not - it's up to the programmer in C to remember to memset(buff, 0, buffsize)15:07
jd__indeed, that'd be calloc()15:07
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities  https://review.openstack.org/8556515:08
*** yfried has joined #openstack-infra15:09
*** yfried__ has quit IRC15:09
*** guitarzan has quit IRC15:09
*** mihgen has joined #openstack-infra15:09
*** dcramer_ has quit IRC15:09
*** wchrisj has quit IRC15:09
*** jcoufal has quit IRC15:10
*** ArxCruz has joined #openstack-infra15:10
*** pdmars has quit IRC15:11
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities  https://review.openstack.org/8556515:11
*** sabari2 has quit IRC15:11
*** branen has quit IRC15:11
*** pdmars has joined #openstack-infra15:12
*** dangers_away is now known as dangers15:13
*** yfried has quit IRC15:13
*** dcramer_ has joined #openstack-infra15:15
fungimordred: i should have said s/expect/hope/, up to the caller to zero blocks allocated and deallocated after all15:15
fungibut as sdague points out vmm should also hopefully prevent leaking between processes even if the author forgot to do so15:16
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt  https://review.openstack.org/8605715:17
*** branen has joined #openstack-infra15:17
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA & Build latest libvirt  https://review.openstack.org/8605715:18
*** maxbit has joined #openstack-infra15:19
mordredfungi: ++15:19
*** andreaf has quit IRC15:20
*** dcramer_ has quit IRC15:20
*** jlibosva has quit IRC15:21
fungiit's considered a security bug in the kernel if it leaks abandoned memory contents from your process into a new process's allocation, just better to be sure anyway if you can15:22
sdaguefungi: also, wouldn't an attempt to exploit this lead to seeing seg faults of processes15:22
sdagueat least at a blanket attack15:22
sdaguewhich the kernel typically logs in dmesg15:22
*** marun has joined #openstack-infra15:22
*** dstanek has quit IRC15:23
*** changbl has joined #openstack-infra15:23
fungisdague: i don't think this would segfault anything. you're reading 64k of surrounding process memory, but it's allocated to that process15:23
sdaguefungi: I thought the point was you could read any 64k15:23
*** alff has joined #openstack-infra15:24
*** alff_ has joined #openstack-infra15:24
fungisdague: if it's been leaked into the that process's memory, then yes. the point that there are circumstances where that *might* happen, and the researchers who announced this said they read passwords and other document material from memory remotely in their tests, but i've also seen a lot of skeptical analysis of those claims15:25
sdaguewell, if it was a web based system, like running owncloud, sure15:25
fungiright now it's zomg-do-whatever-you-can panic, while the more detailed exploit pocs are waiting in the wings to be published15:26
sdaguefungi: gotcha15:26
fungii suspect it's an overreaction, but better safe than sorry15:26
sdaguethis is because the cloudfire folks leaked early before all the patches?15:26
dimshallyn, jamespage, zul, clarkb, jogo - just noticed ibvirt 1.2.2+ from UCA on precise - kicking off a review for it - https://review.openstack.org/#/c/86057/ fyi15:27
jamespagedims, ack15:27
fungisdague: that's the rumor i heard. rh was aware and had apparently given other major distros a heads up yesterday morning, but then it all exploded a few hours later15:27
*** mrodden has joined #openstack-infra15:28
mordredI thought they disclosed after it was announced - and it was the fact taht the distros werent' included on the embargo15:28
sdaguejamespage: what's the eta on the .18 libvirt precise package?15:28
mordredeither way - everything about it was craptastic15:28
zuldims:  okies15:29
*** dcramer_ has joined #openstack-infra15:29
sdaguebecause we're hot patching from our own copy of that package in devstack-gate now15:29
sdagueand I'd like to stop doing that15:29
sdague(if we don't hot patch something like 5% of all tempest runs fail, which means we almost can't land patches)15:29
fungimordred: that was the first thing i heard, then later one of the other not-rh distros piped up and said that rh had started to warn the others, but that the announcement came mere hours on the heels of that because someone leaked it15:29
fungiso i'm guessing they meant to disclose later this week or early next15:30
*** denis_makogon_ has joined #openstack-infra15:30
fungithough i've heard nothing to substantiate that yet, pure speculation on my part15:30
mordredfungi: gah15:31
jamespagesdague, lemme check with hallyn15:31
fungiwe're all still very much in the rumor and misinformation phase of this situation after all15:31
mordreddims: is this the build-libvirt-python-from-pip release?15:32
*** ArxCruz has quit IRC15:32
*** gyee has joined #openstack-infra15:32
*** ArxCruz has joined #openstack-infra15:33
dimsmordred, not sure saw it here - http://javacruft.wordpress.com/2014/04/03/openstack-icehouse-rc1/, we have been bugging UCA team for 1.2.x on precise for a while now and looks like we have it now15:33
sdaguefungi: can I get your assisstance with the right test for - https://review.openstack.org/#/c/84829/7/test-functions.sh - as I'm clearly not quite thinking about this in the same way as you and jeblair15:33
fungisdague: sure, momentarily... just peeled off of a conference call15:34
sdagueyep, no problem15:34
*** ominakov has quit IRC15:37
*** esker has quit IRC15:38
*** msabramo has quit IRC15:40
*** zns has quit IRC15:41
*** homeless has quit IRC15:41
fungisdague: okay, so basically the way the current unit tests in d-g are structured is that you describe a change which is being tested (project name and target branch) with the changes ahead of it in a dependent queue. then you set the branch-affecting variables you would set in a particular job for that change, and see what would have been checked out for various projects which were a part of the15:41
fungiintegration test (some of which may correspond to changes ahead in the queue, others possibly not)15:41
*** signed8bit has quit IRC15:42
*** homeless has joined #openstack-infra15:42
*** signed8bit has joined #openstack-infra15:42
*** che-arne has joined #openstack-infra15:42
openstackgerritBen Nemec proposed a change to openstack-infra/config: Make ironic-undercloud and f20-overcloud jobs non-voting  https://review.openstack.org/8546815:42
bnemecfungi: pleia2: ^Sorry!15:42
bnemecI promise not to change that again. :-)15:43
fungisdague: so for example, you want to test that a nova stable/havana change gets an integration job run with master tempest when you set the project override for tempest to master15:43
bnemecEven if those jobs start passing they obviously have some stability issues so they can stay non-voting for a while.15:43
sdaguefungi: yep15:43
fungisdague: and you may want to confirm that the zuul ref on master for a devstack-gate change ahead of it gets checked out15:43
*** markwash has joined #openstack-infra15:43
fungisdague: and possibly that a neutron change on stable/havana ahead of it causes the zuul ref for that to get checked out rather than the stable/havana branch tip15:44
fungiand so on15:44
sdagueok, sure15:44
*** denis_makogon_ is now known as denis_makogon15:45
*** msabramo has joined #openstack-infra15:46
*** freyes has quit IRC15:46
fungii'm thinking you may want a total of two tests there... one making sure that you can test a change for tempest master against nova stable/havana, and another making sure that you can test a change for nova stabe/havana against tempest master15:46
sdaguefungi: ok15:47
fungiand in the first, use the global override, while in the second use the project-specific override or whatever makes sense15:47
*** jcoufal has joined #openstack-infra15:48
*** zns has joined #openstack-infra15:49
*** jdurgin has quit IRC15:53
sdaguefungi: ok, so that leads me to something like this - https://etherpad.openstack.org/p/zuul-tests15:57
sdaguewhich doesn't pass15:57
*** jdurgin has joined #openstack-infra15:57
jamespagesdague, I've marked it verification done - its had enough testing IMHO - so it should release soonish15:58
sdaguejamespage: great15:59
*** julienvey has quit IRC15:59
jamespagedims, also note that we have a new qemu version coming into icehouse UCA today15:59
*** ArxCruz has quit IRC15:59
sdaguefungi: is there an alphabet issue?15:59
*** ihrachyshka_ has joined #openstack-infra16:00
*** ihrachyshka has quit IRC16:00
fungialphabet soup16:00
*** morgabra has quit IRC16:00
*** morgabra has joined #openstack-infra16:00
sdagueI think I've got a better version of the test, however, it means that python-glanceclient seems to be detected at the wrong branch point16:01
sdagueand I wonder if that's an alphabet issue16:01
*** yfried has joined #openstack-infra16:01
*** alff_ has quit IRC16:01
*** alff has quit IRC16:01
*** matrohon has quit IRC16:02
*** lcostantino has quit IRC16:04
*** blamar has quit IRC16:04
sdaguefungi: any thoughts?16:04
sdagueI'm about out of ideas, and my zuul matrix in my head is clearly at it's limits16:05
*** julim has quit IRC16:05
openstackgerritA change was merged to openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/8581616:05
fungisdague: it's a pyramid16:05
fungisdague: so for the project corresponding to ZA, you'll also have it appearing in ZB, ZC and ZD16:05
fungisdague: and then for the project corresponding to ZB, you'll also have it appearing in ZC and ZD16:05
fungiand so on16:05
sdagueso I have to manually push ZC onto all places?16:06
sdagueregardless of branch?16:07
fungisdague: so if the projects/branches whose changes are being tested are distinct, you'll have four with a ZD ref, three with a ZC ref, two with a ZB ref and one with a ZA ref16:07
fungisdague: test_grenade_backward() is a good demonstrative example of the scenario setup16:07
fungisince it has 5 changes in the queue plus incorporates checkouts of other changes not in the queue16:08
sdagueso is ZB irrespective of branch?16:08
sdagueI guess in my mind I was trying to create the zuul queue16:08
sdaguewhich this is not actually representing16:08
fungiwell, it's representing the merge commits which zuul calculates for the change being tested and the other changes it's possibly being tested with ahead of it16:08
fungiand it's roject+branch-specific16:09
fungibecause zc may be getting tested with a nova master change in zb and a nova stable change in za, so there will be zc refs on nova master and nova stable16:10
fungigrenade, being our best current example of multi-branch testing, wants to be sure that you're testing an upgrade from za to zb when you're running it on a change at zc16:10
*** hogepodge has joined #openstack-infra16:11
funginote that in test_grenade_backward() we have both...16:11
fungiTEST_ZUUL_REFS[keystone]+=' refs/zuul/stable/havana/ZD'16:11
fungiTEST_ZUUL_REFS[keystone]+=' refs/zuul/master/ZD'16:11
*** sdake has joined #openstack-infra16:12
fungibecause (and also for ZE)16:12
fungier, s/because //16:12
*** mihgen has quit IRC16:13
fungibecause zc is a keystone stable/havana change and zd is a keystone master change16:13
fungiand ze needs to be tested with those16:14
*** e0ne_ has quit IRC16:14
*** derekh has quit IRC16:14
*** e0ne has joined #openstack-infra16:14
*** ociuhandu_ has joined #openstack-infra16:15
*** moted has joined #openstack-infra16:17
*** markmcclain has joined #openstack-infra16:17
*** ociuhandu_ has quit IRC16:17
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Support Matrix Tie Parent Plugin in wrappers  https://review.openstack.org/8564216:18
*** jgallard has quit IRC16:18
fungilogstash job queue splodeyness16:18
*** e0ne has quit IRC16:19
*** ihrachyshka_ has quit IRC16:19
fungicluster status is yellow16:20
*** krtaylor has quit IRC16:21
*** hogepodge has quit IRC16:21
fungiseveral unassigned shards and more being replicated at the moment16:22
msabramohashar: thoughts on https://review.openstack.org/84778 ?16:22
fungithe es cluster and all the workers look like they're still online16:22
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides  https://review.openstack.org/8482916:24
sdaguefungi: so how about that?16:24
*** jcoufal has quit IRC16:25
*** Sukhdev has joined #openstack-infra16:26
hasharmsabramo: gotta leave sorry :/16:29
*** blamar has joined #openstack-infra16:29
*** zns has quit IRC16:29
hasharmsabramo: haven't reviewed that patch yet :-]16:30
*** hashar has quit IRC16:30
*** alff_ has joined #openstack-infra16:30
*** alff has joined #openstack-infra16:30
*** markmc has quit IRC16:31
*** blamar has quit IRC16:32
*** blamar has joined #openstack-infra16:32
msabramomgagne: Wondering if you have any thoughts on https://review.openstack.org/8477816:34
fungisdague: lookin16:34
mgagnemsabramo: reading16:34
*** timrc is now known as timrc-afk16:34
*** dizquierdo has quit IRC16:34
*** dstanek has joined #openstack-infra16:34
*** saju_m has joined #openstack-infra16:34
*** jcoufal has joined #openstack-infra16:35
*** krtaylor has joined #openstack-infra16:39
*** sabari has joined #openstack-infra16:40
*** MIDENN_ has quit IRC16:40
*** mdenny has joined #openstack-infra16:40
*** ildikov_ has quit IRC16:41
*** MarkAtwood has joined #openstack-infra16:42
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added parallelization options  https://review.openstack.org/7551416:42
*** yassine has quit IRC16:42
*** bhuvan has joined #openstack-infra16:45
*** MarkAtwood has quit IRC16:46
openstackgerritSean Dague proposed a change to openstack-infra/config: update config variable to OVERRIDE_foo_PROJECT_BRANCH  https://review.openstack.org/8609016:48
sdaguefungi: also, with the var change, I need that ^^^^^16:48
fungiahh, right16:50
*** jroll has quit IRC16:50
*** jroll has joined #openstack-infra16:51
annegentlettx: or anyone: is there a rollup location in launchpad to get a count of all blueprints implemented across icehouse for ceilometer cinder glance horizon keystone neutron nova swift trove16:52
annegentlettx: seems like the /openstack project only contains -infra and docs16:53
annegentlettx: I can count by hand I guess16:53
*** MarkAtwood has joined #openstack-infra16:54
annegentlefungi: or clarkb any ideas for a blueprint count across 9 projects? ^^16:54
fungiannegentle: combine nine queries? (that's what i'd probably end up doing)16:55
fungithen again, i'm a brute16:55
annegentlefungi: ha. brute.16:55
annegentlefungi: yeah 36+ browser tabs was the best I came up with16:56
*** jgrimm has quit IRC16:56
fungi(i tend to use brute-force solutions to infrequent problems because they can end up taking less time that devising a more elegant solution)16:56
annegentletotally get that16:57
fungier go http://xkcd.com/1205/16:57
fungier, ergo16:58
annegentlefungi: ttx probably does this task often tho16:58
annegentlebut it's pycon week.16:58
annegentlethat should be on that chart LOL16:58
koolhead17might have it16:58
fungioh, mining http://status.openstack.org/release/16:59
fungigood suggestion koolhead1716:59
koolhead17annegentle:  ^^16:59
*** krotscheck has quit IRC17:00
*** mriedem has joined #openstack-infra17:00
fungihow quickly i seem to forget that exists17:00
*** krotscheck has joined #openstack-infra17:00
koolhead17fungi: your getting old :P17:00
fungitell me about it17:00
ttxannegentle: yes, what fungi said17:00
fungito be perfectly fair, that was koolhead17's suggestion, because i'm dense17:01
*** bhuvan_ has joined #openstack-infra17:01
annegentlekoolhead17: thanks!17:02
*** julim has joined #openstack-infra17:02
*** andreykurilin_ has joined #openstack-infra17:02
openstackgerritA change was merged to openstack-infra/config: update config variable to OVERRIDE_foo_PROJECT_BRANCH  https://review.openstack.org/8609017:02
marunfungi: is it possible for me to be personally notified if the neutron functional job fails?17:02
*** _nadya_ has joined #openstack-infra17:03
*** amcrn has quit IRC17:04
*** bhuvan has quit IRC17:04
*** mrmartin has quit IRC17:04
*** rfolco has quit IRC17:05
*** fifieldt has quit IRC17:05
*** ihrachyshka has joined #openstack-infra17:05
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat auth bug 1304544  https://review.openstack.org/8609317:06
uvirtbotLaunchpad bug 1304544 in heat "test_server_cfn_init fails in setup with Unauthorized" [Undecided,New] https://launchpad.net/bugs/130454417:06
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Only run jobs with neutron on neutron changes  https://review.openstack.org/8484017:07
*** amotoki has quit IRC17:07
fungimarun: i think what you'll want to do is make sure you have gerrit set to watch the projects it runs on and send you e-mail for all comments, then filter the message bodies looking for the job name/failure pattern and stick those messages somewhere you'll notice17:07
fungimarun: i do something similar to separate different sorts of gerrit e-mails into different buckets so i can keep tabs on things better17:08
*** rfolco has joined #openstack-infra17:09
*** timrc-afk is now known as timrc17:09
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs  https://review.openstack.org/8317217:09
*** jgrimm has joined #openstack-infra17:09
*** jgrimm has quit IRC17:10
*** jgrimm has joined #openstack-infra17:11
*** andreykurilin_ has quit IRC17:11
*** harlowja has joined #openstack-infra17:16
mtreinishfungi, clarkb: any thoughts on getting this pushed through?: https://review.openstack.org/#/c/85431/17:18
*** SumitNaiksatam has quit IRC17:18
clarkbmtreinish: this week is going to be a bad week17:19
mtreinishclarkb: ok, np it can wait then17:20
mtreinishclarkb: what's this week?17:21
*** jpich has quit IRC17:21
clarkbmtreinish: heartbleed17:21
*** _nadya_ has quit IRC17:22
*** morganfainberg_Z is now known as morganfainberg17:22
dstufftdhellmann: FWIW as I udnerstand it the namespace bug is basically unresolvable in pip17:23
*** Ajaeger1 has joined #openstack-infra17:24
*** sdague has quit IRC17:27
*** sdague has joined #openstack-infra17:29
*** Sukhdev has quit IRC17:30
*** e0ne has joined #openstack-infra17:31
*** jerryz has joined #openstack-infra17:32
dimsjamespage, fyi, ran into this -r/--arch requirement in euca command line https://eucalyptus.atlassian.net/browse/DOC-807 http://logs.openstack.org/57/86057/2/check/check-grenade-dsvm/f459067/logs/old/devstacklog.txt.gz17:32
dimsjamespage, thanks for the heads up about qemu version17:32
marunfungi: danke17:33
*** amcrn has joined #openstack-infra17:35
*** Ryan_Lane has joined #openstack-infra17:35
Ajaeger1clarkb: the translation jobs are working fine with your patch as far as I could see when I checked earlier today. Could you review https://review.openstack.org/#/c/85973/ as followup, please?17:35
jerryzHI, anybody knows whether i can use DependentPipelineManager on Check pipeline? What if only two changes from two different projects tested together can pass?17:35
clarkbjerryz: you can. it may not always work as expected though17:36
clarkbAjaeger1: right now, still dealing with heartbleed fallout17:36
clarkbAjaeger1: infra is probably going to fall behind this week especially with people at pycon17:36
fungiAjaeger1: also, what is an obsolete translation entry?17:36
*** ianw has quit IRC17:36
Ajaeger1clarkb: I see.17:36
Ajaeger1fungi: those are the entries that caused us troubles. These are commented out entries.17:37
Ajaeger1https://review.openstack.org/#/c/85958/ for one patch where obsolete entries were removed17:37
fungiAjaeger1: aha, so entries where there is no msgid because the original text is no longer present in the software17:38
Ajaeger1fungi: exactly17:39
Ajaeger1fungi: thanks for the review17:40
openstackgerritDan Prince proposed a change to openstack-infra/reviewday: Quit smoking.  https://review.openstack.org/8610417:41
*** fbo_away has quit IRC17:42
*** fbo_away has joined #openstack-infra17:43
*** bhuvan_ has quit IRC17:43
*** fbo_away is now known as fbo17:43
jerryzclarkb: thanks. In my scenario, the two changes heavily depend on each other, they can not pass on one's own, only Z=C1+C2 can pass. will DependentPiplelineManager still apply?17:43
*** SumitNaiksatam has joined #openstack-infra17:43
clarkbjerryz: pipeline managers apply to all changes though17:43
fungijerryz: what you're really wanting is cross-project dependencies, i suspect... a feature we've been discussing for a while17:46
fungijerryz: unless you're talking about changes on the same project and branch, in which case just make them depend on each other in gerrit and zuul will handle that case automatically17:46
*** jcoufal has quit IRC17:46
*** MarkAtwood has quit IRC17:47
jerryzfungi: cross-project dependency it is17:47
fungijerryz: that's definitely a feature we want, but we don't currently have a way to support it17:48
*** _nadya_ has joined #openstack-infra17:48
jerryzfungi: what do you do know? like manually force one change to merge and test the other?17:48
fungijerryz: though there has been a fair amount of design work around it, no implementation work has been started yet as far as i know17:48
fungijerryz: we make sure changes are implemented in pieces so they can be correctly serialized17:49
*** esker has joined #openstack-infra17:49
*** Sukhdev has joined #openstack-infra17:50
*** e0ne has quit IRC17:50
fungijerryz: even when we eventually implement cross-project dependencies, we still will require them to be able to be tested and landed in series. allowing intertwined commits where neither can work unless both are merged supports very bad design habits17:50
*** resker has joined #openstack-infra17:50
*** e0ne has joined #openstack-infra17:51
jerryzfungi: a seamless switch-over ?17:51
fungijerryz: right, you need to be able to land support in one project, then land the code which uses it in a different project, support proper deprecation of features and configuration, et cetera17:52
jerryzfungi: got it. i will see if we can serialize in my scenario.17:52
fungijerryz: otherwise you break people doing continuous deployment17:52
*** esker has quit IRC17:53
fungijerryz: just like you wouldn't want to land multiple changes in one project where the first breaks the software and then the next one fixes it. proper testing already prevents that... now we're just talking about how to spread the serialization properly between multiple projects17:53
*** hogepodge has joined #openstack-infra17:55
*** e0ne has quit IRC17:55
*** resker has quit IRC17:55
*** Guest44106 is now known as Adri200017:55
*** gokrokve has quit IRC17:55
dhellmanndstufft: can it be resolved somewhere else, like setuptools, or is this a case of allowing multiple ways to do something causing fundamental incompatibilities?17:56
*** anteaya has quit IRC17:56
dstufftdhellmann: I'm not super knowledgeable about this issue, it long predates my involvement, but my understanding is there are multiple ways which are incompatible. It might be possible for setuptools to resolve those incompatibilities I dunno17:56
dstufftor they might just be fundamentally incompat17:56
dhellmanndstufft: ok17:57
*** e0ne has joined #openstack-infra17:57
*** zns has joined #openstack-infra17:58
*** jamielennox|away is now known as jamielennox17:58
*** zns has quit IRC17:58
*** melwitt has joined #openstack-infra17:58
*** zns has joined #openstack-infra17:59
*** mspreitz has joined #openstack-infra17:59
dhellmanndstufft: it's not really clear how much this affects us with sdague's devstack changes, which is part of why I wanted to start the conversation17:59
sdagueyeh, now that we are punting on -e for oslo libs, devstack will stop being odd for folks18:00
dhellmannsdague: I think we'll have a period where existing systems need to be cleaned up, but yeah, I hope that change helps a lot18:03
*** esker has joined #openstack-infra18:03
*** thedodd has quit IRC18:03
*** MarkAtwood has joined #openstack-infra18:04
*** che-arne has quit IRC18:04
*** thedodd has joined #openstack-infra18:06
*** sandywalsh_ has joined #openstack-infra18:07
*** sandywalsh has quit IRC18:09
*** wchrisj has joined #openstack-infra18:10
*** zns has quit IRC18:11
*** freyes has joined #openstack-infra18:12
*** thuc has joined #openstack-infra18:13
*** zns has joined #openstack-infra18:13
*** bhuvan has joined #openstack-infra18:15
*** thuc_ has quit IRC18:16
*** thuc has quit IRC18:17
*** dcramer_ has quit IRC18:19
*** UtahDave has joined #openstack-infra18:20
sdagueyeh, I left the clean on olso at least18:20
sdaguewe could add the same clean logic for other packages18:20
*** dcramer_ has joined #openstack-infra18:21
*** mdenny has quit IRC18:23
*** mdenny has joined #openstack-infra18:23
*** hogepodge has quit IRC18:26
*** _nadya_ has quit IRC18:26
*** mspreitz_ has joined #openstack-infra18:28
*** mspreitz has quit IRC18:29
*** SumitNaiksatam_ has joined #openstack-infra18:29
*** SumitNaiksatam has quit IRC18:29
*** SumitNaiksatam_ is now known as SumitNaiksatam18:29
*** mdenny has quit IRC18:30
*** mdenny has joined #openstack-infra18:30
*** mspreitz has joined #openstack-infra18:31
*** mspreitz_ has quit IRC18:32
*** mspreitz_ has joined #openstack-infra18:34
*** dcramer_ has quit IRC18:35
*** persia has quit IRC18:35
*** mspreitz has quit IRC18:36
*** mspreitz_ is now known as mspreitz18:36
*** 23LAAAIJP has joined #openstack-infra18:36
*** dcramer_ has joined #openstack-infra18:37
*** 23LAAAIJP has quit IRC18:37
*** 23LAAAIJP has joined #openstack-infra18:37
*** 23LAAAIJP is now known as persia18:37
*** thuc_ has joined #openstack-infra18:37
*** atiwari has joined #openstack-infra18:39
*** saju_m has quit IRC18:44
*** yolanda has quit IRC18:47
*** thuc has joined #openstack-infra18:50
*** thedodd has quit IRC18:50
*** ildikov_ has joined #openstack-infra18:51
*** thedodd has joined #openstack-infra18:52
*** thuc_ has quit IRC18:53
*** thuc has quit IRC18:54
pabelangerfungi, do you have access to the rackspace mycloud interface? I'd be curious to see how many of your images are in a building (0%) / error state.18:56
*** dkehn_ has joined #openstack-infra18:56
*** yolanda has joined #openstack-infra18:56
*** weshay has quit IRC18:57
*** ihrachyshka has quit IRC18:57
fungipabelanger: i do, but don't have time to jump in there and look just now18:58
clarkbpabelanger: and three is the the open question of whether nor not it is currently safe to log in to it18:58
pabelangerfungi, clarkb: understood18:58
fungiclarkb: well, this would be the jenkins tenant, which nodepool is constantly authenticating to18:58
sdagueclarkb or jeblair: https://review.openstack.org/#/c/84829/ the branch override piece would be nice to land if we could, unit tests seem solid now18:59
fungiso that ship has probably sailed (but maybe not if their dashboard and api endpoint are individually vulnerable in separate ways)18:59
clarkbcan any rackspace folks address that? if we log in to the web ui and/or talk to openstack api endpoints at rackspace are we 1. vulnerable to credential leakage and 2. if not because patching has happened are we vulnerable to mitm because ssl certs haven't been swapped out?18:59
clarkbphschwartz: ^ I think you mentioned you are at rax18:59
fungikeeping tabs on https://community.rackspace.com/general/f/34/t/3596 too19:00
fungialso, meeting time19:00
clarkband https://status.rackspace.com/ but none of them address the problem direclty as it pertains to me logging in and doing stuff19:00
phschwartzclarkb: I can pass the question along as I am not sure of the answer19:00
*** derekh has joined #openstack-infra19:00
clarkbphschwartz: thanks19:01
*** weshay has joined #openstack-infra19:02
clarkbphschwartz: I suppose there is a third option where we were never vulnerable because heartbeat wasn't used or older openssl was used. But concrete info would be handy19:02
*** ianw has joined #openstack-infra19:02
*** yolanda has quit IRC19:04
*** esker has quit IRC19:07
*** maxbit has quit IRC19:10
*** gokrokve has joined #openstack-infra19:13
clarkbphschwartz: o/19:14
phschwartzclarkb: It looks like it is option 3. Our F5's are running a version of openssl that is not affected.19:15
phschwartzall of the openstack api's are behind LBs, but other api's in our infra are still being verified to be behind LBs or that they do not have an affected openssl version.19:15
clarkbphschwartz: what about web dashboard?19:16
*** cody-somerville has quit IRC19:16
*** dangers is now known as dangers_away19:16
phschwartzSorry don't know which dashboard your talking about. There are a lot that we have. mind passing a url (or are you talking about the mycloud.rackspace.com)19:17
*** weshay has quit IRC19:17
clarkbphschwartz: ya whatever it is I get when I log in to spin up a server19:18
clarkbor add a dns record19:18
*** ihrachyshka has joined #openstack-infra19:18
fungiphschwartz: "https://mycloud.rackspace.com/"19:18
phschwartzThose were tested and don't appear to have the vuln either (actually looks like a majority of our things don't have heartbeat enabled)19:20
clarkbphschwartz: great, thank you for the info19:23
phschwartzclarkb: np19:23
*** jcoufal has joined #openstack-infra19:24
*** david_lyle_ has joined #openstack-infra19:26
*** rcleere has quit IRC19:26
*** gyee has quit IRC19:28
*** dklyle has joined #openstack-infra19:28
mordredclarkb: for HP: https://community.hpcloud.com/status/incident/253319:29
*** stevebaker has quit IRC19:30
*** stevebaker has joined #openstack-infra19:30
*** david-lyle has quit IRC19:30
clarkbmordred: thanks19:31
*** dklyle has quit IRC19:31
clarkbso I think we are good to update things via rax and hpcloud api/dashboards19:31
clarkbfungi: ^19:31
*** dklyle has joined #openstack-infra19:31
fungiclarkb: agreed19:31
*** mrmartin has joined #openstack-infra19:32
fungithough i may need to get an early dinner right after the meeting and start beating on it as soon as i get back from that19:32
*** david_lyle_ has quit IRC19:32
clarkbsame here19:32
clarkbI am running on fumes19:33
*** ildikov has joined #openstack-infra19:33
fungilast night was sorta late for me ;)19:33
*** jswarren has joined #openstack-infra19:35
*** malini is now known as malini_afk19:36
*** thedodd has quit IRC19:37
phschwartzwhat time is the meeting?19:38
phschwartzI want to get more involved and I figure that is one of the steps19:38
clarkbphschwartz: now, sorry I should've mentioned it earlier19:38
*** thedodd has joined #openstack-infra19:38
clarkbphschwartz: we are just wrapping up though so we can get back to dealing with heartbleed, but feel free to join19:39
clarkband a log will be posted at http://eavesdrop.openstack.org19:39
phschwartzah, I joined, but will look at the log and will make sure to be there next week19:39
phschwartzso the meeting is at 3 est? I thought it was at 4. my mistake defn19:40
*** ociuhandu has quit IRC19:40
*** dklyle is now known as david-lyle19:40
clarkbphschwartz: it is scheduled for 1900 UTC so depending on DST it moves19:41
phschwartzThats what got me. lol19:41
*** anteaya has joined #openstack-infra19:41
*** jeremyb has quit IRC19:41
*** jeremyb has joined #openstack-infra19:41
fungiphschwartz: pretty much all openstack team meetings are scheduled based on utc, since we've got people participating worldwide19:43
*** thedodd has quit IRC19:43
*** Ryan_Lane1 has joined #openstack-infra19:43
*** markmc has joined #openstack-infra19:44
clarkbfungi: mordred: I am going to run really quickly and grab a sandwich then get back to the fun19:44
fungiclarkb: okay, thanks19:45
*** alff__ has joined #openstack-infra19:47
*** zns has quit IRC19:47
*** Ryan_Lane1 has quit IRC19:48
*** alff has quit IRC19:49
*** Sukhdev has quit IRC19:50
*** mspreitz has quit IRC19:50
*** zns has joined #openstack-infra19:51
ianwis there anything useful I can do to help the devstack/fedora job we mentioned?19:51
*** zns has quit IRC19:52
fungiianw: since the tripleo cloud already has fedora nodes, you might talk to pleia2 about the possibility of having us run a simple devstack job there as a proving ground, perhaps in the check-tripleo or experimental-tripleo pipeline initially while rh is working out the quota issues on the second tripleo region they've built19:56
*** zns has joined #openstack-infra19:57
ianwfungi: ok, will do19:57
jogodims: https://bugs.launchpad.net/nova/+bug/130410719:58
uvirtbotLaunchpad bug 1304107 in qemu "Libvirt error launching instance - Device 'virtio-net-pci' could not be initialized" [High,Fix released]19:58
jogoohh it was fixed since last night, looks like it was libvirt19:58
fungijogo: yep, dims and jamespage were discussing earlier today in here19:59
jogoneato that was quick19:59
anteayaianw: she is in transit today to pycon, but you can leave a message for her or email20:00
fungijogo: at least i think that's why they were talking about libvirt 1.2.2 in uca for precise, but maybe that was an unrelated coincidence20:01
Ajaeger1fungi, anteaya  I'm considering to take a graph or two from the presentations at http://docs.openstack.org/infra/publications. Is that ok? What kind of credit do I need to give?20:01
fungiAjaeger1: i think there should be a license in that repository/branch20:02
*** derekh has quit IRC20:02
Ajaeger1fungi: not visible on the website, let me check the repo...20:03
*** andreykurilin_ has joined #openstack-infra20:03
fungiAjaeger1: yeah, i'm not finding it either20:03
fungithat may be a major oversight on our part20:03
Ajaeger1fungi: can't find it in the repo either - just checked two recent branches20:05
fungiAjaeger1: i think the intent was that it should be apache 2 or cc-by 3 but i'm not positive20:05
fungiAjaeger1: please open a bug on that against openstack-ci, since we definitely should correct it asap20:05
anteayasince the point of having them up is for people to use them20:06
fungiAjaeger1: i think it's safe to work on the assumption that it would use the same licensing as other content on docs.o.o (so apache or cc-by) but we as authors need to have a conversation confirming that so we can correct it properly i think20:06
Ajaeger1fungi: ok, will add a slide at the end and reference where I've taken the graphs from. Thanks.20:09
* fungi disappears to find early dinner, and then get back to the fun20:09
Ajaeger1fungi: enjoy!20:09
Ajaeger1bug reported at https://bugs.launchpad.net/openstack-ci/+bug/1304636 in case any of you wants to give it some quick comments20:10
uvirtbotLaunchpad bug 1304636 in openstack-ci "Copyright information missing for publications" [Undecided,New]20:10
dimsjogo, take a quick peek at the uca/precise here - https://review.openstack.org/#/c/86057/ - one bug popped out of it - https://bugs.launchpad.net/devstack/+bug/1304571. jamespage mentioned that a fresh qemu will hit the archive today.20:14
uvirtbotLaunchpad bug 1304571 in devstack "euca-bundle-image requires a new architecture parameter" [Undecided,In progress]20:14
*** jp_at_hp1 has joined #openstack-infra20:16
*** ociuhandu has joined #openstack-infra20:17
*** jp_at_hp has quit IRC20:17
jogothat is caused by a new version of euca2ools?20:19
jogosdague: re https://review.openstack.org/8549620:19
ianwfungi: so just chatting with dprince, he's not opposed to the idea of using ooo cloud for a smaller devstack testing job, but his preference would probably be to be using RAX20:19
jogohow do I drop hacking for g-r?20:20
ianwfungi: if we're running a smaller job just for devstack changes, can it run in the single provider of RAX?20:20
*** stevebaker has quit IRC20:20
*** stevebaker has joined #openstack-infra20:20
jogoclarkb: ^20:21
*** mdenny has quit IRC20:21
*** mdenny has joined #openstack-infra20:21
*** mrmartin has quit IRC20:23
*** sweston has joined #openstack-infra20:23
openstackgerritJay Faulkner proposed a change to openstack-infra/reviewstats: Fix capitalization of my gerrit username  https://review.openstack.org/8614220:24
clarkbianw: as long as it doesn't gate20:26
clarkbianw: the gate piece is sort of the distinguishing criteria20:26
ianwclarkb: yeah, i mean we wouldn't want it gating for some time until it proved it stability anyway20:27
sdaguejogo: oh, you need the ability to punch through the mirror20:29
sdagueafter tc meeting I can ponder20:30
*** dcramer_ has quit IRC20:30
*** dprince has quit IRC20:31
*** e0ne has quit IRC20:31
*** e0ne has joined #openstack-infra20:32
ianwclarkb / fungi : so in terms of getting this done (sorry for the neophyte questions) the first thing would be a separate jenkins job in the job-builder?20:32
clarkbianw: first thing would be a nodepool image and flavor for fedora nodes20:33
clarkbianw: then a JJB job20:33
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for grenade g-api service start failure bug 1304652  https://review.openstack.org/8614520:34
uvirtbotLaunchpad bug 1304652 in grenade "g-api did not start" [Undecided,New] https://launchpad.net/bugs/130465220:34
jogosdague: sounds good20:35
*** e0ne has quit IRC20:35
ianwclarkb: how do people usually develop on nodepool?  it's not clear to me how to setup a sandbox20:35
*** dcramer_ has joined #openstack-infra20:35
clarkbianw: if you have credentials for a cloud you can set one up. However it is probably sufficient to just run the steps manually on said clouds base image20:36
clarkbianw: eg you don't need to test nodeppol as much as you have to test the image builds20:36
ianwclarkb: ok, cool, i'll look into that20:37
ianwclarkb: when i've verified that, i'll probably start a thread on openstack-infra, sound good?20:38
*** vhoward has joined #openstack-infra20:39
*** freyes has quit IRC20:40
*** jp_at_hp1 has quit IRC20:41
*** gyee has joined #openstack-infra20:41
*** jp_at_hp has joined #openstack-infra20:41
*** stevebaker has quit IRC20:42
*** stevebaker has joined #openstack-infra20:42
*** Ajaeger1 has quit IRC20:43
*** dcramer_ has quit IRC20:44
*** dcramer_ has joined #openstack-infra20:46
*** eharney has quit IRC20:49
*** hogepodge has joined #openstack-infra20:49
*** ihrachyshka has quit IRC20:54
*** e0ne has joined #openstack-infra20:54
*** e0ne has quit IRC20:56
*** e0ne has joined #openstack-infra20:56
*** westmaas has quit IRC20:57
*** mrda_away is now known as mrda20:57
*** david-lyle is now known as david-lyle_afk20:57
*** mrda is now known as Guest5285120:58
*** ihrachyshka has joined #openstack-infra20:58
*** e0ne has quit IRC20:59
*** Guest52851 is now known as mrda21:02
*** ruhe2 has joined #openstack-infra21:03
*** zns has quit IRC21:03
*** dkliban has quit IRC21:03
*** ruhe2 has left #openstack-infra21:04
*** aysyd has quit IRC21:08
*** ihrachyshka has quit IRC21:08
sdagueclarkb: got a few minutes for https://review.openstack.org/#/c/84829/ ? we've only got about 7 days until we'd be setting the tempest branch, so I'd really like to be able to get this exercised before then21:09
clarkbsdague: I feel like if I start making exceptions to do stuff that isn't #heartbleed that is a bad thing21:11
sdagueclarkb: ok, fair21:11
jogosdague: so now that TC is over how do  take hacking out of g-r21:11
sdaguewell, as soon as you come up for air, it would be appreciated21:11
sdaguejogo: the global requirements job itself can punch the mirror21:12
*** jpeeler has quit IRC21:12
sdagueso we should figure out what it needs to do to do that21:12
*** jp_at_hp has quit IRC21:13
jogosdague: so that will allow us to pull packages from pypi.python.org?21:14
asselinHi, what version of Jenkins is openstack-infra using?21:18
sdaguejogo: that's my thought21:18
clarkbasselin: it should tell you if you visit a jenkins server21:18
*** salv-orlando_ has joined #openstack-infra21:19
jogoclarkb: ^ does that make sense21:19
sdaguebecause I don't think you actually want to require pyflakes upgrade on all the projects before new hacking21:19
sdaguewhich is what's going to happen21:19
jogosdague: clarkb had a different idea. but I am with you on this one sdague21:19
jogosdague: well only projects that pin pyflakes21:19
jogowhich is trove and oslo-incubator21:19
asselinclarkb, thanks got it21:19
jogoand patches up to fix both21:20
clarkbright no one should pin it directly21:20
jogoclarkb: but people do21:20
clarkbinstead should rely on hacking21:20
sdaguejogo: so they specify it?21:20
jogosdague: https://review.openstack.org/#/c/67138/21:20
sdaguewait, if everyone's supposed to rely on hacking, why are these in the list at all?21:20
clarkbStevenK: hey, re heartbleed any idea of whether or not launchpad is safe now?21:20
jogosdague: because hacking needs them ;)21:20
clarkbStevenK: or any idea where I should check? #launchpad on freenode?21:21
sdagueyeh, but I'd much rather remove them from g-r, let hacking punch the mirror, then not allow people to specify them21:21
jogosdague: ++21:21
*** salv-orlando has quit IRC21:21
*** salv-orlando_ is now known as salv-orlando21:21
sdaguethough is there a transitive dep mirror problem?21:21
clarkbhttps://identi.ca/launchpadstatus is quiet21:22
sdagueclarkb: I guess we don't have a way to say "put this in the mirror, but don't let anyone use it"21:22
clarkbsdague: nope, which is why I am a proponent of decoupling the mirror from enforcement21:22
*** denis_makogon has quit IRC21:22
sdagueyeh, I agree21:22
jogoso moving forward what does that mean for hacking?21:26
dhellmannfungi: did you have a chance to look at the updates to https://review.openstack.org/#/c/85487/ ?21:26
mordredjogo, sdague: we _USED_ to need the specific pins per project - I imagine those are just hysterical raisins21:27
*** dkranz has quit IRC21:27
sdaguemordred: they used to use pep8 directly21:28
*** mfer has quit IRC21:28
sdagueeveryone is going through hacking now21:28
*** ameade has quit IRC21:28
sdaguewhich I think is a better place to control those versions21:28
mordredit is21:28
mordredthat's the idea at least21:28
sdagueI think trove and oslo-incubator are the only exceptions, and I think those are easily fixed21:29
dhellmannsdague: yeah, I don't see us calling pyflakes directly in the incubator21:29
sdaguemordred: so I guess the question here is how to get the mirror to have these packages even if not in g-r21:30
sdaguefrom where we currently stand21:30
*** reed has quit IRC21:30
sdaguethats where my understanding of infra falls down21:30
clarkbsdague: you cannot21:31
clarkbyou could possibly do a supplemental list, but currently we use the mirror for enforcement21:31
SlickNiksdague: trove's using hacking as well. There's this patch in flight to address the issue that jogo just brought up https://review.openstack.org/#/c/67138/3/test-requirements.txt21:32
sdagueclarkb: supplemental list would be fine21:32
sdagueclarkb: got an entry point for me tot try to propose something?21:32
*** dachary has quit IRC21:32
clarkbsdague: the mirror jobs in openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config21:33
clarkbsdague: they should pass a reqs file to the run mirror script, if you add another file that should work21:33
*** dachary has joined #openstack-infra21:35
SlickNiksdague / jogo: Trove should be good once that patch merges; correct?21:35
sdagueclarkb: ok, I'm trying to figure where they pass a file, I just see them passing a git tree21:36
jogoSlickNik: yup21:36
clarkbsdague: oh it may just be a tree21:37
clarkbsdague: then it looks for files in that tree?21:37
*** jooools has quit IRC21:37
sdagueclarkb: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/pypi-mirror.yaml21:38
sdaguethat's where I'm dead ending21:38
*** pdmars has quit IRC21:39
clarkbsdague: ok, I think it will look for files that look like requirements. You may need to add anothe rproject to the list21:39
fungidhellmann: i started looking at it earlier before you had jobs passing. need to revisit but i think as long as you have some mechanical solution to ensuring parity/symmetry then my concerns from the previous iteration (about not being able to adequately review its sanity) are covered21:39
sdagueclarkb: I suppose we could always add mirror-requirements.txt to g-r21:40
*** dizquierdo has joined #openstack-infra21:40
dhellmannfungi: yeah, the latest version has a little tool for doing that check -- you have to have all of openstack checked out to use the tool, but I did :-)21:40
sdaguemaye that would pick that up?21:40
fungidhellmann: yep, saw, just didn't finish covering yet what with all the other distractions, but i think this is probably safe until we find better routes21:41
*** sweston has quit IRC21:43
*** david_lyle_ has joined #openstack-infra21:44
*** david_lyle_ is now known as david-lyle21:44
*** david-lyle is now known as david_lyle21:44
*** russellb has quit IRC21:44
*** thomasem has quit IRC21:45
*** russellb has joined #openstack-infra21:45
*** david_lyle_ has joined #openstack-infra21:46
*** reed has joined #openstack-infra21:46
*** david-lyle_afk has quit IRC21:47
*** dcramer_ has quit IRC21:48
openstackgerritSean Dague proposed a change to openstack/requirements: add mirror requirements list  https://review.openstack.org/8617121:48
*** dcramer_ has joined #openstack-infra21:49
*** david_lyle has quit IRC21:50
*** mriedem has left #openstack-infra21:52
*** ihrachyshka has joined #openstack-infra21:52
*** e0ne has joined #openstack-infra21:54
*** hogepodge has quit IRC21:55
*** e0ne has quit IRC21:56
*** e0ne has joined #openstack-infra21:56
*** andreykurilin_ has quit IRC21:58
*** eharney has joined #openstack-infra21:59
*** andreykurilin_ has joined #openstack-infra21:59
*** markmcclain has quit IRC22:00
*** markmcclain has joined #openstack-infra22:01
*** e0ne has quit IRC22:01
*** markmcclain has quit IRC22:02
*** gondoi is now known as zz_gondoi22:02
*** markmcclain has joined #openstack-infra22:02
*** russellb has quit IRC22:03
*** markmcclain has quit IRC22:03
*** markmcclain has joined #openstack-infra22:03
*** russellb has joined #openstack-infra22:03
*** gokrokve_ has joined #openstack-infra22:04
SergeyLukjanovsdague, probably I just should sleep a bit, but I have a crazy idea22:04
sdagueSergeyLukjanov: I like crazy :)22:04
SergeyLukjanovsdague, mirror-requirements == [pip install global-requirements then pip freeze minus global-requirements]22:05
SergeyLukjanovI mean all transient requirements22:05
*** hogepodge has joined #openstack-infra22:06
sdagueSergeyLukjanov: yeh, that could be22:06
SergeyLukjanovsdague, on the other side, why do we need list of them is we'll install them anyway22:06
*** dtroyer has quit IRC22:07
SergeyLukjanovsdague, oh, I see your explanation about versions22:07
*** andreykurilin_ has quit IRC22:07
*** gokrokve has quit IRC22:07
*** dcramer_ has quit IRC22:10
*** dtroyer has joined #openstack-infra22:10
*** dizquierdo has quit IRC22:12
*** markmc has quit IRC22:13
*** andreaf has joined #openstack-infra22:13
*** dstanek has quit IRC22:13
anteayamattoliverau: morning22:15
anteayamattoliverau: I'm trusting you got your ballot question answered?22:15
anteayaI didn't see an email in my inbox from you22:15
mattoliverauanteaya: sorry, not yet, thanks to the heartbleed openssl bug, yesterday became a bit of a race to patch all the servers, and then got sidetracked will other work that was then behind. I think apart of the problem was I, for some reason was dropped from the dev list. Re-subscribed the other day. So getting email again I wonder if I missed the ballot email.22:19
*** Sukhdev has joined #openstack-infra22:20
andreafsdague: ping22:22
sdagueandreaf: what's up? I'm about to drop for a few22:23
andreafsdague: nothing urgent - I made the devstack change for domain_name here when you have a sec https://review.openstack.org/#/c/86023/22:23
*** sabari has quit IRC22:24
*** sabari has joined #openstack-infra22:24
sdagueandreaf: is there an extra _ in the 2nd change line?22:25
sdagueat the end of the var22:25
sdagueI think I see another issue actually22:25
andreafsdague: I'll remove the _ thanks22:26
sdagueI think there are 2 small issues22:26
sdagueI just put it in the review22:26
sdaguethen I'm +222:26
sdagueok, off for a bit22:26
andreafsdague: thanks22:27
mgagneI think I broke gerrit, got 500 after submitting my review =(22:28
mgagnenow it's back22:29
*** jergerber has quit IRC22:30
*** bhuvan has quit IRC22:31
mattoliverauanteaya: email sent :)22:32
*** markmcclain has quit IRC22:33
*** markmcclain has joined #openstack-infra22:33
clarkbmgagne: sorry for the turbulence22:35
*** adalbas has quit IRC22:35
mgagneclarkb: =)22:36
*** prad has quit IRC22:37
fungimgagne: *i* broke gerrit22:38
fungi(by stopping it to reset database account passwords)22:38
*** gokrokve_ has quit IRC22:38
fungiawesome of you to notice me trying to be sneaky ;)22:39
*** homeless has quit IRC22:39
*** nkinder has quit IRC22:40
fungitoday's disruption brought to you by the letters h and i, and by the number heartbleed22:40
*** gokrokve has joined #openstack-infra22:43
*** Ryan_Lane1 has joined #openstack-infra22:46
*** andreaf has quit IRC22:46
*** andreaf has joined #openstack-infra22:47
* mgagne watches fungi22:47
*** adalbas has joined #openstack-infra22:47
fungimgagne: do your worst. my government was watching me long before you even thought of it22:48
* fungi makes obscene gestures at the nsa22:49
*** atiwari has quit IRC22:49
*** dims has quit IRC22:49
*** thuc has joined #openstack-infra22:50
*** msabramo has quit IRC22:50
*** Ryan_Lane1 has quit IRC22:50
*** melwitt has quit IRC22:52
*** david_lyle_ has quit IRC22:55
*** thuc has quit IRC22:55
*** e0ne has joined #openstack-infra22:55
*** yamahata has quit IRC22:58
*** e0ne has quit IRC22:58
StevenKclarkb: I daresay it is, but either there, or #canonical-sysadmin22:58
*** changbl has quit IRC23:00
anteayamattoliverau: your status with the -dev ml does not effect the status of the ballot23:01
*** zehicle_at_dell has joined #openstack-infra23:02
* jhesketh catches up on meeting minutes23:04
jheskeththe infra meeting is now 5am for me :-(23:04
StevenKThe TripleO meeting used to be at 3am for me, but I'm not sure what it is now with daylight savings23:05
anteayajhesketh: :(23:05
jheskethlets just move to a flat world with 1 timezone ;-)23:05
anteayawas brief today, fungi chaired and wanted to get back to fixing all the things23:06
*** msabramo has joined #openstack-infra23:06
anteayaI had rebooted my weechat server and it didn't allow me ssh access until the last few minutes of the meeting23:06
anteayajhesketh: sounds exciting23:06
anteayagot anyplace in mind?23:06
*** MarkAtwood has quit IRC23:07
*** MarkAtwood has joined #openstack-infra23:07
*** jcoufal has quit IRC23:08
fungii am notorious for wanting to fix things23:08
anteayayou are23:09
*** dkranz has joined #openstack-infra23:09
jheskethheh, nice23:11
phschwartzfungi: blasphemy23:16
*** wenlock has quit IRC23:16
phschwartzhacky work arounds are all that are allowed ;)23:16
fungiphschwartz: after all, the universe perpetually tends toward an ever increasing state of entropy?23:18
*** mriedem1 has joined #openstack-infra23:19
phschwartzfungi: See, someone who sees things my way23:22
*** ihrachyshka_ has joined #openstack-infra23:29
*** thuc has joined #openstack-infra23:32
*** ihrachyshka has quit IRC23:32
*** thuc_ has joined #openstack-infra23:33
*** msabramo1 has joined #openstack-infra23:35
*** nkinder has joined #openstack-infra23:36
*** thuc has quit IRC23:36
*** msabramo has quit IRC23:37
*** rlandy has quit IRC23:38
*** markmcclain has quit IRC23:40
*** markmcclain has joined #openstack-infra23:40
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add example of comment formatting  https://review.openstack.org/8593023:41
*** UtahDave has quit IRC23:42
*** flaper87 is now known as flaper87|afk23:43
*** dstanek has joined #openstack-infra23:44
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add note on ci.openstack.org source  https://review.openstack.org/8593323:45
*** mriedem1 has quit IRC23:47
openstackgerritSean Dague proposed a change to openstack-dev/pbr: make pbr use hacking directly  https://review.openstack.org/8618923:48
*** dteselkin_ has quit IRC23:49
*** dteselkin_ has joined #openstack-infra23:49
mordredsdague: does that work ^^ ?23:51
mordredsdague: we did the other thing before because hacking depends on pbr and circular dependency23:51
sdaguemordred: tox -e pep8 did23:51
mordredI support the idea23:52
sdagueso beyond that, I make no guaruntees23:52
sdaguethat, however, is what was blocking this - https://review.openstack.org/#/c/86171/23:52
*** gokrokve has quit IRC23:52
*** e0ne has joined #openstack-infra23:55
*** derekh has joined #openstack-infra23:56
*** yamahata has joined #openstack-infra23:58
*** mrda is now known as mdavies23:58
*** mdavies is now known as mrda23:59
*** msabramo1 has quit IRC23:59
*** e0ne has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!