Wednesday, 2014-07-02

jheskethclarkb: in the interest of moving things forward, do you think it'd be better to just trial jenkins07 and if it fails undo it?00:00
clarkbjhesketh: yes, and I don't expect it to be a problem because gearman deals with those env vars directly00:01
*** unicell has joined #openstack-infra00:01
clarkbjhesketh: to me the tricky bit were the changes you already wrote and I approved to make the job not use the plugin00:01
*** wenlock has quit IRC00:01
jheskethclarkb: okay, when's a good time to do this?00:01
clarkbjhesketh: I can probably take a stab at that tomorrow00:02
clarkbjhesketh: this evening isn't good beacuse EMELTING00:02
clarkband I am already babysitting a pvmove and image rebuilds00:02
jheskethall good00:03
jheskeththanks clarkb :-)00:03
*** david-lyle has quit IRC00:05
*** mmaglana has quit IRC00:05
*** david-lyle has joined #openstack-infra00:06
*** msabramo has quit IRC00:06
clarkbmordred: I don't know why is skipping review.o.o00:06
*** anode has quit IRC00:06
clarkbmordred: any chance you know why that is happening?00:06
*** chuckC has quit IRC00:07
arosenclarkb: after this merged: I'm not able to push to:  ssh:// yet00:07
arosenI guess it might take a little time?00:07
clarkbarosen: yes the project hasn't been created for some reason00:07
*** atiwari has quit IRC00:07
clarkbarosen: because puppet hasn't run on review.openstack.org00:07
arosenah okay.00:07
jesusaurusis new? how does that get triggered?00:07
clarkbits the thing I am hoping mordred will look into above because its breaking me now too00:07
clarkbjesusaurus: its new ish been around for a few weeks. and cron00:07
clarkbI really wish puppet agent --disable was a bit more clear about whether or not it is disabled00:08
*** msabramo has joined #openstack-infra00:08
jeblairclarkb: puppet agent --disable is dangerous on any git-related machines because of the ordering00:09
jeblairclarkb: that's why i'm updating the project rename instructions to indicate that the run_all cron should be stopped00:10
clarkbjeblair: ya  Ithink I merged that change00:10
jeblair(there is not a good way to do that now, we need to make that better)00:10
clarkbjeblair: but I don't even know if puppet is disabled00:10
clarkball I know is it has been 48ish minutes since puppet ran on review.o.o00:10
*** david-lyle has quit IRC00:10
clarkband mordred was the last person to report working on this stuff00:10
jeblairclarkb: check syslog?00:10
*** zehicle_at_dell has quit IRC00:11
clarkbsyslog only shows the 2319 catalog run complete message00:11
clarkbpuppet is running on other nodes on a 15 minutes interval00:12
clarkbmaybe I should just try running it manually with --noop. maybe error reporting is broken? /me does this00:12
jeblairclarkb: it may not be running on review.o.o if it's breaking on a git host00:12
clarkbjeblair: oh00:13
clarkbalso --noop on review.o.o was clean /me checks git0100:13
melwittclarkb: what does it mean to add the hudson-openstack account to the nova bug team? (
uvirtbotLaunchpad bug 1326503 in python-novaclient "hudson-openstack not working for python-novaclient" [Undecided,New]00:13
clarkbpuppet on failed with return code: 1 thats it00:13
clarkbmelwitt: its a launchpad groups membership permissions thing00:14
clarkbmelwitt: some projects have oepn bug and allow gerrit to assign bugs with no issues00:15
clarkbmelwitt: but if the project isn't open then you need to explicitly give the user gerrit uses permissions to do operations like that00:15
clarkberr: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: File[/etc/httpd/conf/httpd.conf]00:15
clarkbI think we must've udpated apache module?00:15
clarkbmaybe I should've been meaner with my vote on that...00:15
melwittclarkb: okay. I was trying to see if I could add it but I don't see a way. I'm not a admin of nova-bugs but I am a member00:16
sdaguehmmm... the requirements fix didn't seem to do a thing -
clarkbsdague: we don't enforce requirements on unittests00:17
clarkbsdague: so the update has to merge first00:17
clarkbjesusaurus: ^ any chance you have time to debug the apache moduelthing?00:17
sdagueclarkb: which update?00:17
*** melwitt has quit IRC00:17
clarkbsdague: the requirements file update in oslo-incubator00:17
*** dangers is now known as dangers_away00:18
jeblairclarkb, mordred: okay, so we've been trying to adopt this policy where we approve potentially disruptive puppet patches when their authors are around to monitor and fix changes00:19
jeblairclarkb, mordred: we don't seem to actually be doing that00:19
jesusaurusclarkb: sure, did that "revert" get merged?00:20
jesusaurusah, so it did00:20
clarkbjeblair: yes I dunno who approved it or when00:20
jesusaurusclarkb: do you have a link of the error for me?00:20
clarkbbut I failed at -1ing so :(00:20
jeblairmordred: approved it00:21
clarkbjesusaurus only what I pasted above. puppetboard may have more00:21
jeblairand neither monitored it himself nor mode sure that jesusaurus was around to do so :(00:21
mtreinishclarkb: yes, I'd support pacman everywhere :)00:22
*** msabramo has quit IRC00:22
marunin job yaml, I'm assuming there is an 80 char limit?  how would you suggest I format the following?00:22
marunsudo -H -u stack OS_SUDO_TESTING=1 OS_ROOTWRAP_CMD='sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf' tox -e functional00:22
jeblairmarun: i'm not aware of such a limit -- however, you might consider making a new tox env00:23
jeblairmarun: maybe 'sudo-functional'00:23
marunjeblair: ah, fair enough.  putting the configuration in the neutron tree likely makes more sense than in config00:24
*** Ryan_Lane has joined #openstack-infra00:24
*** gangil has quit IRC00:24
*** gokrokve has quit IRC00:24
jeblairyeah, seems like a good place for those vars00:24
*** dims__ has quit IRC00:25
lifelessnow I have to call a variable pseudo-functional somewhere that sudo is used.00:25
jeblairlifeless: all of our software is sudo-functional ;)00:26
ianwmordred: re what environment do you use for local testing?00:28
jeblairianw: i think a change in that series made a doc update about how to run it?00:29
ianwjeblair: i assume ubuntu trusty?00:31
clarkbianw: tools/ is what you want00:32
jesusaurusclarkb: okay, it looks like the two templates (one from the cgit module and one from the upgraded apache module) have some significant differences00:33
jesusaurusand theres no simple way to just turn off the one in the apache module00:33
jesusaurusso we should probably revert back to 0.0.400:33
*** Ryan_Lane has quit IRC00:37
jeblairianw: well, i hope it works on precise since that's what nodepool is running on right now :)00:39
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Revert "Revert "Downgrade puppetlabs-apache to version 0.0.4.""
*** signed8bit is now known as signed8bit_ZZZzz00:40
*** HACKING-TWITTER has quit IRC00:41
*** dims__ has joined #openstack-infra00:43
mordredjeblair, clarkb: sorry - my bad - I failed to flag that in my brain as potentially disruptive.00:43
*** HACKING-TWITTER has joined #openstack-infra00:43
mordredalthough in retrospect, how I could have possibly thought that was non-disruptive is beyond me00:44
jeblairmordred: 'revert' something seems like a nice safe thing to do, but really it should have been "try upgrading apache module again"00:44
*** Ryan_Lane has joined #openstack-infra00:44
jeblairmordred: or "revert revert apache upgrade"00:45
jeblairwe're basically one revert off00:45
*** msabramo has joined #openstack-infra00:45
*** jp_at_hp has quit IRC00:45
openstackgerritMaru Newby proposed a change to openstack-infra/config: Fix neutron func job to use gate-specific tox env
*** signed8bit_ZZZzz is now known as signed8bit00:46
*** jdahlin has joined #openstack-infra00:46
*** signed8bit is now known as signed8bit_ZZZzz00:46
*** signed8bit_ZZZzz is now known as signed8bit00:46
lifelessis the requirements job broken or something?00:49
lifelessI'm seeing patches from 'longgeek' that seem to do exactly what the requirements jobs do00:49
lifelesse.g. 10339000:49
*** dstanek is now known as dstanek_zzz00:50
*** yaguang has joined #openstack-infra00:51
* jesusaurus begins the 30 minute commute home00:53
clarkbpvmove complete00:54
*** xianghui has joined #openstack-infra00:58
*** bdpayne has quit IRC00:59
*** nati_ueno has quit IRC00:59
*** nati_ueno has joined #openstack-infra01:01
*** talluri has joined #openstack-infra01:06
harlowjaanyone aware of a bug for 'Routes is in an unsupported or invalid wheel' issue?01:07
clarkbharlowja: yes they have a bad wheel01:08
clarkbharlowja: a requirements change was pushed and merged to remove that version01:08
harlowjakk, thx01:08
*** jp_at_hp has joined #openstack-infra01:08
harlowjaguess thats still filtering out to the other projects, thx clarkb01:09
*** nati_ueno has quit IRC01:09
*** nati_ueno has joined #openstack-infra01:10
sdagueharlowja: you need to sync manually for unit tests01:13
harlowjasdague ya, makes sense, just seeing alot of oslo-incubator reviews hitting that; already up01:13
sdagueharlowja: yep, I rebased my change on that one01:14
harlowjasdague kk, good idea01:14
*** jp_at_hp has quit IRC01:14
*** dstanek_zzz is now known as dstanek01:17
*** nelsnelson has joined #openstack-infra01:19
*** Sukhdev has quit IRC01:20
*** Sukhdev has joined #openstack-infra01:22
*** marcoemorais has quit IRC01:22
*** nelsnelson has quit IRC01:23
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support for negative requirements
*** nelsnelson has joined #openstack-infra01:24
*** signed8bit is now known as signed8bit_ZZZzz01:25
*** otherwiseguy has quit IRC01:27
*** moted has quit IRC01:27
*** otherwiseguy has joined #openstack-infra01:27
*** xchu has joined #openstack-infra01:31
*** zz_dimtruck is now known as dimtruck01:32
mordredoh my. ^^01:32
mordredjhesketh: what crazy things are you doing now?01:32
jheskethmordred: trying to earn a beer off jogo_away01:33
jheskethmordred: the idea is to not retrigger automatic checks when a reviewer has left a -101:33
mordredjhesketh: nicely done. that makes me want to actually implement that feature in zuul - so taht you could ask zuul how many beers each person is owed01:33
mordredjhesketh: nice01:34
*** nelsnelson has quit IRC01:34
jheskethmordred: and a count system that stops code from merging until the beer has been claimed01:34
mordredjhesketh: if we start now, we could have it ready by next april fool's01:35
*** yaguang has quit IRC01:36
StevenKI don't know if I like reviewing turning into a drinking game ...01:37
clarkbStevenK: embrace the liver damage01:38
clarkbjhesketh: if you ever make it to portland I have kegs >_>01:38
jheskethclarkb: you only say that because of how many drinks I owe you ;-)01:38
jheskethoh shiny! :-)01:38
*** Ryan_Lane has quit IRC01:38
clarkbjhesketh: maybe I will just make you help me brew the next beer to fill the kegs while we drink the ones already kegged01:39
*** MarkAtwood has joined #openstack-infra01:39
jheskethI'm all up for learning :-)01:39
*** markwash has quit IRC01:42
ianwmordred / anyone : is this image build script cloning* ?01:43
ianwit seems to be downloading *a lot*01:44
mordredianw: yse01:46
mordredianw: it clones every single git repo on git.openstack.org01:46
mordredianw: however, it does local caching01:46
mordredianw: so once you build once, the _second_ time you build will go MUCH quicker01:46
ianwmordred: hmm, do you know how big that will end up being?01:46
mordrednot off the top of my head, no01:46
ianwlike 1gb or more like 10?01:47
* mordred looks01:47
mordredianw: the resulting image will be 2G01:48
lifelessFWIW my image-create cache is 13G01:48
mordredianw: in the process, it will produce 2.5G of cache01:48
lifelessbut I've been building many different things01:48
lifelessand have lots of different root dirs etc01:49
mordredlifeless: you're crazy01:49
*** nosnos has joined #openstack-infra01:49
ianwok, thanks, i'm just in the land of 200gb/month isp quotas01:49
lifelessmordred: foxlike01:49
mordredianw: AH. ewww01:49
lifelessianw: ok so this is why we cache things :)01:49
*** gokrokve has joined #openstack-infra01:49
lifelessianw: so you don't go over quota01:49
ianwlifeless: yeah, but i've still got to get it the first time :)01:50
*** baohua has joined #openstack-infra01:50
*** yaguang has joined #openstack-infra01:51
*** otherwiseguy has quit IRC01:51
* ianw super-checks i'm not accidentally tethered, because that would probably bankrupt me01:52
*** mriedem has quit IRC01:53
*** praneshp has quit IRC01:54
*** r1chardj0n3s is now known as r1chard_afk01:54
*** nosnos has quit IRC01:55
*** nosnos has joined #openstack-infra01:55
*** otter768 has joined #openstack-infra01:56
*** homeless has quit IRC01:57
*** dkranz has quit IRC01:59
*** hdd_ has joined #openstack-infra02:02
*** wenlock has joined #openstack-infra02:03
ianw$ git clone git://
ianwCloning into 'python-congressclient'...02:04
ianwfatal: The remote end hung up unexpectedly02:04
ianw^ is this just me?  it seems pretty repeatable02:04
*** dstufft|laptop has joined #openstack-infra02:06
*** signed8bit_ZZZzz is now known as signed8bit02:06
*** signed8bit is now known as signed8bit_ZZZzz02:06
*** nati_ueno has quit IRC02:07
clarkbianw: its not just you02:08
clarkbpuppet isn't running on the git farm right now02:08
*** nati_ueno has joined #openstack-infra02:10
*** otherwiseguy has joined #openstack-infra02:12
ianwclarkb: is that a known issue?02:14
clarkbya it came up in scrollback around 0000UTC02:16
clarkbI am currently melting so trying to avoid hot computer with keys to do more fixing02:16
*** MarkAtwood has quit IRC02:18
*** nati_ueno has quit IRC02:21
*** nati_ueno has joined #openstack-infra02:21
*** Ryan_Lane has joined #openstack-infra02:21
*** zhiyan_ is now known as zhiyan02:22
*** nati_ueno has quit IRC02:22
*** xianghui has quit IRC02:24
*** dstanek is now known as dstanek_zzz02:26
*** arnaud has quit IRC02:29
*** zhiyan is now known as zhiyan_02:30
*** signed8bit_ZZZzz is now known as signed8bit02:31
*** r1chard_afk is now known as r1chardj0n3s02:31
*** signed8bit has quit IRC02:32
*** signed8bit has joined #openstack-infra02:32
*** signed8bit has quit IRC02:32
*** talluri has quit IRC02:34
*** camunoz is now known as camunoz_away02:36
*** zhiyan_ is now known as zhiyan02:42
*** wenlock has quit IRC02:43
*** changbl has joined #openstack-infra02:43
*** chuckC has joined #openstack-infra02:44
*** penguinRaider has quit IRC02:44
*** pballand has joined #openstack-infra02:45
*** Sukhdev has quit IRC02:46
*** dstanek_zzz is now known as dstanek02:48
*** alexandra has joined #openstack-infra02:49
*** alexandra is now known as asettle02:49
*** pballand has quit IRC02:49
*** harlowja is now known as harlowja_away02:50
*** praneshp has joined #openstack-infra02:52
*** asettle has quit IRC02:54
*** otherwiseguy has quit IRC02:58
*** ArxCruz has joined #openstack-infra02:59
*** yaguang has quit IRC03:00
*** yaguang has joined #openstack-infra03:01
*** nati_ueno has joined #openstack-infra03:02
*** theannegentle has joined #openstack-infra03:03
*** otherwiseguy has joined #openstack-infra03:04
*** theannegentle has quit IRC03:04
*** xianghui has joined #openstack-infra03:05
*** xchu has quit IRC03:08
*** dims__ has quit IRC03:09
*** gyee has quit IRC03:10
*** alexandra_ has joined #openstack-infra03:11
*** alexandra_ is now known as asettle03:11
*** mwagner_ has quit IRC03:15
*** r1chardj0n3s is now known as r1chardj0n3s_afk03:20
*** zhiyan is now known as zhiyan_03:22
*** zhiyan_ is now known as zhiyan03:24
*** zehicle_at_dell has joined #openstack-infra03:26
*** mwagner_ has joined #openstack-infra03:27
*** nosnos has quit IRC03:27
*** zhiyan has quit IRC03:30
*** camunoz_away is now known as camunoz03:33
*** gangil has joined #openstack-infra03:35
*** zhiyan has joined #openstack-infra03:37
*** gothicmindfood_ has joined #openstack-infra03:37
*** r1chardj0n3s_afk is now known as r1chardj0n3s03:37
*** zhiyan is now known as zhiyan_03:39
*** otherwiseguy has quit IRC03:39
*** otherwiseguy has joined #openstack-infra03:40
*** zhiyan_ is now known as zhiyan03:40
*** zhiyan is now known as zhiyan_03:46
*** talluri has joined #openstack-infra03:48
*** zhiyan_ is now known as zhiyan03:50
*** arnaud has joined #openstack-infra03:50
*** zehicle_at_dell has quit IRC03:56
*** zehicle_at_dell has joined #openstack-infra03:57
*** praneshp_ has joined #openstack-infra04:01
*** asettle has quit IRC04:01
*** pcrews has quit IRC04:02
*** zehicle_at_dell has quit IRC04:02
*** zehicle_at_dell has joined #openstack-infra04:02
*** lcheng has joined #openstack-infra04:02
*** praneshp has quit IRC04:04
*** praneshp_ is now known as praneshp04:04
*** ArxCruz has quit IRC04:06
*** zehicle_at_dell has quit IRC04:07
*** zehicle_at_dell has joined #openstack-infra04:08
*** garyh has quit IRC04:09
*** yfried_ has quit IRC04:10
*** nelsnelson has joined #openstack-infra04:11
*** olaph has quit IRC04:12
*** Sukhdev has joined #openstack-infra04:12
*** gothicmindfood_ has quit IRC04:21
openstackgerritJamie Lennox proposed a change to openstack/requirements: Add requests-mock package to global requirements
*** alexandra_ has joined #openstack-infra04:30
*** alexandra_ is now known as asettle04:31
*** dimtruck is now known as zz_dimtruck04:31
*** r1chardj0n3s is now known as r1chardj0n3s_afk04:32
*** crc32 has quit IRC04:32
*** r1chardj0n3s_afk is now known as r1chardj0n3s04:36
*** dims__ has joined #openstack-infra04:37
*** nosnos has joined #openstack-infra04:42
*** dims__ has quit IRC04:42
*** arnaud has quit IRC04:47
*** arnaud has joined #openstack-infra04:49
*** yfried_ has joined #openstack-infra04:53
*** otter768 has quit IRC04:55
*** colettecello has joined #openstack-infra04:55
*** gothicmindfood has quit IRC04:56
*** HACKING-TWITTER has quit IRC05:02
*** dims__ has joined #openstack-infra05:02
*** gangil has quit IRC05:03
*** dstanek is now known as dstanek_zzz05:04
*** asettle has quit IRC05:06
*** dims__ has quit IRC05:07
*** otherwiseguy has quit IRC05:08
*** colettecello is now known as gothicmindfood05:08
*** HACKING-TWITTER has joined #openstack-infra05:11
*** alexandra_ has joined #openstack-infra05:11
*** otherwiseguy has joined #openstack-infra05:15
*** marun is now known as marun_afk05:16
*** gothicmindfood has left #openstack-infra05:16
*** Ryan_Lane has quit IRC05:21
*** amitpp has joined #openstack-infra05:21
*** otherwiseguy has quit IRC05:21
*** afazekas_ has joined #openstack-infra05:21
*** HACKING-TWITTER has quit IRC05:22
*** rwsu has quit IRC05:24
*** basha has joined #openstack-infra05:25
*** zul has quit IRC05:27
*** ildikov has joined #openstack-infra05:27
*** cody-somerville has quit IRC05:27
*** cody-somerville has joined #openstack-infra05:27
*** zul has joined #openstack-infra05:28
*** gothicmindfood has joined #openstack-infra05:28
*** HACKING-TWITTER has joined #openstack-infra05:29
openstackgerritChris Buccella proposed a change to openstack/requirements: Update glanceclient to 0.13.1
*** bdpayne has joined #openstack-infra05:33
*** gema_ is now known as gema05:33
*** amitpp has quit IRC05:35
*** basha has quit IRC05:37
*** otherwiseguy has joined #openstack-infra05:38
*** chandan_kumar has joined #openstack-infra05:38
*** basha has joined #openstack-infra05:38
*** HACKING-TWITTER has quit IRC05:40
*** amitpp has joined #openstack-infra05:40
*** lcheng has quit IRC05:41
*** otherwiseguy has quit IRC05:42
*** basha_ has joined #openstack-infra05:45
*** basha has quit IRC05:46
*** basha_ is now known as basha05:46
*** otherwiseguy has joined #openstack-infra05:48
*** mrmartin has joined #openstack-infra05:49
*** dstanek_zzz is now known as dstanek05:51
*** bdpayne has quit IRC05:52
*** otherwiseguy has quit IRC05:52
*** ildikov has quit IRC05:53
*** hdd_ has quit IRC05:58
*** zz_dimtruck is now known as dimtruck05:59
*** dstanek is now known as dstanek_zzz06:01
*** rcarrillocruz has joined #openstack-infra06:02
*** dims__ has joined #openstack-infra06:03
*** nati_ueno has quit IRC06:03
*** rcarrill` has quit IRC06:04
*** HACKING-TWITTER has joined #openstack-infra06:04
*** dstanek_zzz is now known as dstanek06:06
*** dims__ has quit IRC06:08
*** akuznetsova has joined #openstack-infra06:09
*** mrmartin has quit IRC06:10
*** Sukhdev has quit IRC06:11
clarkbFYI re gerrit release "There is a major regression in 2.8.6 that causes exceptions in database transactions."06:13
StevenKclarkb: ... Yay?06:14
clarkbStevenK: it probably won't affect us06:14
clarkbbecause we don't upgrade quickly06:14
clarkbbut there was talk at one point of maybe upgraded soonish06:14
*** dstanek is now known as dstanek_zzz06:15
*** HACKING-TWITTER has quit IRC06:16
akuznetsovaHi all, I updated requirement in one repository due to global-requirements from master branch and got this
*** nati_ueno has joined #openstack-infra06:16
akuznetsovaCan somebody explain me what the problem is?06:16
clarkbakuznetsova: ideally you shouldn't depend on pep8 and flake8 directly. instead hacking will pull it what it needs06:17
clarkbakuznetsova: I believe the problem is the version specified there is different that what hacking specifies06:17
clarkbor flake806:18
akuznetsovashould I just remove pep8 and flake8 from my requirements?06:18
clarkbyes I think so06:18
akuznetsovaclarkb, thanks, I will try06:19
*** jdahlin has quit IRC06:27
*** gokrokve has quit IRC06:29
*** HACKING-TWITTER has joined #openstack-infra06:32
*** rcarrill` has joined #openstack-infra06:34
*** ildikov has joined #openstack-infra06:35
*** rcarrillocruz has quit IRC06:36
*** dimtruck is now known as zz_dimtruck06:37
*** rfolco has quit IRC06:38
*** ildikov has quit IRC06:40
*** r1chardj0n3s is now known as r1chardj0n3s_afk06:41
*** HACKING-TWITTER has quit IRC06:47
*** doude has joined #openstack-infra06:47
*** alexandra_ has quit IRC06:47
*** dstufft|laptop has quit IRC06:49
*** nati_ueno has quit IRC06:51
*** tkelsey has joined #openstack-infra06:51
openstackgerritValeriy Ponomaryov proposed a change to openstack-infra/config: Added devstack job for manila
*** GheRivero_afk is now known as GheRivero06:56
*** gokrokve has joined #openstack-infra07:00
*** pcrews has joined #openstack-infra07:01
*** rdopieralski has joined #openstack-infra07:02
*** gokrokve_ has joined #openstack-infra07:02
openstackgerritA change was merged to openstack-infra/config: Fixing deprecation warnings
*** gokrokv__ has joined #openstack-infra07:04
*** gokrokve has quit IRC07:04
*** mrmartin has joined #openstack-infra07:06
*** gokrokve_ has quit IRC07:07
*** gokrokv__ has quit IRC07:08
*** rdopieralski has quit IRC07:11
*** mrda is now known as mrda_away07:11
*** pcrews has quit IRC07:17
*** ociuhandu has joined #openstack-infra07:19
*** flaper87|afk is now known as flaper8707:19
*** jlibosva has joined #openstack-infra07:24
*** mrmartin has quit IRC07:24
*** afazekas_ is now known as afazekas07:25
*** otherwiseguy has joined #openstack-infra07:26
*** andreykurilin_ has joined #openstack-infra07:28
*** mzoeller has joined #openstack-infra07:29
*** arnaud has quit IRC07:31
*** otherwiseguy has quit IRC07:31
*** otherwiseguy has joined #openstack-infra07:32
*** hashar has joined #openstack-infra07:33
mattoliveraunight all07:34
mattoliverauits dinner making time!07:34
*** flaper87 is now known as flaper87|afk07:35
*** otherwiseguy has quit IRC07:37
*** pblaho has joined #openstack-infra07:38
openstackgerritRadoslav Gerganov proposed a change to openstack-infra/config: Add button that shows/hides CI comments in Gerrit
*** jcoufal has joined #openstack-infra07:38
*** praneshp has quit IRC07:38
openstackgerritA change was merged to openstack-infra/nodepool: Update pbr version
rgerganovmordred, are you around?07:42
*** mkerrin has joined #openstack-infra07:45
*** camunoz is now known as camunoz_gone07:47
*** Hal_ has joined #openstack-infra07:50
*** ildikov has joined #openstack-infra07:50
*** mestery has quit IRC07:51
*** andreykurilin_ has quit IRC07:53
*** lcheng has joined #openstack-infra07:53
*** basha has quit IRC07:54
*** basha has joined #openstack-infra07:55
*** mestery has joined #openstack-infra07:55
*** jpich has joined #openstack-infra07:56
*** penguinRaider has joined #openstack-infra07:57
*** markmc has joined #openstack-infra08:00
tteggelhpcloud have a nova deploy going on this morning GMT. starting in 30 mins. expect some api calls to fail.08:00
hasharrgerganov: mordred has been idle for 2h30m, he is most probably sleeping by now08:03
hasharSergeyLukjanov: <tteggel> hpcloud have a nova deploy going on this morning GMT. starting in 30 mins. expect some api calls to fail.08:04
hasharSergeyLukjanov: I think you are the only one with access at this time of the day :D08:04
rgerganovhashar, I am looking for someone that can make a very simple change on review-dev.openstack.org08:04
rgerganovhashar, do you know anyone?08:04
*** gokrokve has joined #openstack-infra08:04
*** dims__ has joined #openstack-infra08:06
hasharrgerganov: if you are logged in with your launchpad account you can browse the All-Projects rights,access08:06
hasharI can't list members of the Administrators group though08:07
SergeyLukjanovhashar, hey08:08
hasharrgerganov: I guess anyone that did a change recently most probably has admin rights. You might want to ask on the infra mailing list.  But maybe everything is puppetized and you can do the change in openstack-infra/config.git08:09
*** gokrokve has quit IRC08:09
rgerganovhashar, thanks;  I am looking for someone that can modify files on the server; specifically
hasharSergeyLukjanov: I guess hpcloud would no more respond to nodepool requests for a little bit. No clue about the impact :D08:10
SergeyLukjanovhashar, I think that it shouldn't break anything08:10
hasharrgerganov: apparently not in puppet. So I have no clue :/08:10
*** dims__ has quit IRC08:11
SergeyLukjanovwe've implemented some mechanisms for non-responding clouds before enabling tripleo cloud08:11
*** IvanBerezovskiy has joined #openstack-infra08:11
hasharsounds good :-]08:12
*** yfried_ has quit IRC08:12
*** yfried_ has joined #openstack-infra08:13
*** yfried_ has joined #openstack-infra08:13
*** amitpp has quit IRC08:14
*** oomichi has quit IRC08:14
*** amitpp has joined #openstack-infra08:14
*** fifieldt_ has joined #openstack-infra08:16
*** flaper87|afk is now known as flaper8708:17
*** yfried_ has quit IRC08:18
*** yfried has joined #openstack-infra08:18
*** fifieldt has quit IRC08:18
*** ihrachyshka has joined #openstack-infra08:19
openstackgerritNikita Konovalov proposed a change to openstack-infra/infra-specs: StoryBoard search spec
*** guangjun has joined #openstack-infra08:20
*** lcheng has quit IRC08:21
*** basha has quit IRC08:26
*** Hal_ has quit IRC08:26
*** jp_at_hp has joined #openstack-infra08:26
*** lcheng has joined #openstack-infra08:26
*** Hal_ has joined #openstack-infra08:26
*** gokrokve has joined #openstack-infra08:30
*** fifieldt_ is now known as fifieldt08:31
*** otherwiseguy has joined #openstack-infra08:32
openstackgerritA change was merged to openstack-infra/config: Adds a docs-specs repository
*** gokrokve has quit IRC08:34
*** Hal_ has quit IRC08:35
*** otherwiseguy has quit IRC08:36
*** derekh_ has joined #openstack-infra08:41
*** yfried has quit IRC08:42
*** dizquierdo has joined #openstack-infra08:43
*** MIDENN_ has quit IRC08:47
*** mdenny has joined #openstack-infra08:47
*** jcoufal has quit IRC08:48
*** jgallard has joined #openstack-infra08:48
*** Hal_ has joined #openstack-infra08:50
*** mrmartin has joined #openstack-infra08:53
openstackgerritA change was merged to openstack-infra/config: Index the os-collect-config logs in tripleo jobs
*** pelix has joined #openstack-infra08:56
*** gema has quit IRC09:00
*** e0ne has joined #openstack-infra09:00
*** penguinRaider has left #openstack-infra09:02
*** gema has joined #openstack-infra09:02
*** gokrokve has joined #openstack-infra09:02
*** Longgeek has joined #openstack-infra09:03
*** NikitaKo_ has joined #openstack-infra09:03
*** Longgeek has quit IRC09:03
*** gokrokve has quit IRC09:03
*** gokrokve has joined #openstack-infra09:04
*** Longgeek has joined #openstack-infra09:04
*** dims__ has joined #openstack-infra09:07
*** gokrokve has quit IRC09:09
*** dims__ has quit IRC09:12
*** zz_johnthetubagu is now known as johnthetubaguy09:17
*** tcammann_ has joined #openstack-infra09:18
tcammann_Hey the gate-oslo-incubator-python33 job has been failing consistently for the last 8 hours or so. e.g. Any ideas?09:19
openstackgerritChristian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger
*** andreaf has joined #openstack-infra09:24
markmctcammann_, looks like pycrypto isn't building09:25
*** maxbit has quit IRC09:25
therveI think it's more failing with the Routes dependency09:25
openstackgerritChristian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger
pelixhashar: any idea as to why the bug in mina-sshd wouldn't be ticked if using a host key generated with openssl?09:27
hasharpelix: glad you ask: I have no clue :-(09:27
pelixshould we push to have the hash mismatch patch added in the mean time to git-review or wait until a fixed version of gerrit appears?09:28
hasharpelix: I reproduced the issue with some old version of Gerrit I had.  I am fighting to upgrade the Gerrit with the version that might fix the issue09:29
hasharhow is Gerrit installed in the git-review integration tests ?09:29
hasharGerrit Code Review 2.8.1-4-ga1048ce ready09:29
hashar \O/09:29
pelixdownloaded from
openstackgerritChristian Hofstädtler proposed a change to openstack-infra/jenkins-job-builder: Add Gitlab Merge Request trigger
hasharpelix: Christian (our Gerrit/Java guru) cherry picked a few patches and crafted me
hasharmight want to mirror that to some place and try running tests with it09:33
hasharthat fix the hash mismatch issue for me09:33
openstackgerritSergey Skripnick proposed a change to openstack-infra/devstack-gate: Add option to enable profiling
hasharmight cause over bugs though (we never know)09:33
openstackgerritSergey Skripnick proposed a change to openstack-infra/devstack-gate: Add option to enable profiling
*** ch has joined #openstack-infra09:38
tcammann_markmc: Looks like the wheel package was updated yesterday09:38
*** liusheng is now known as liusheng-leave09:39
chhi! i've got technical trouble signing the icla on review.o.o -> Server Error/09:39
chCannot store contact information; what am i doing wrong?09:40
*** MIDENN_ has joined #openstack-infra09:40
*** mdenny has quit IRC09:40
hasharpelix: our bug report will be updated with bit more details ( )09:44
pelixhashar: going to try testing with gerrit-2.8.6 since David Pursehouse did a release this morning09:45
pelixor :) rather09:46
hasharpelix: might not have the mina sshd patch though09:46
*** ch has left #openstack-infra09:46
pelixit does, but apparently it also has a DB regression, nuts09:46
tcammann_I believe the gate-oslo-incubator-python33 is failing due to the wheel releasing v0.23.0 yesterday and the Routes is now for some reason invalid. We should change the requirements to use 0.22.0 or below.09:47
tcammann_Routes uses the .whl format09:48
pelixhashar: hopefully he should have a new release sometime in the next day or two that fixes the DB regression, so for git-review it should be as simple as pointing at the new release.09:48
hasharpelix: awesome! Looking forward to the tests finally passing reliably09:51
*** MIDENN_ has quit IRC09:51
derekh_In the last hour a commit merged so that tripleo os-collect-config logs are indexed by logstash, but the "message" is being populated with %{logmessage} instead of actual data, has anybody got any idea why ?
*** mdenny has joined #openstack-infra09:52
*** mkoderer has quit IRC09:52
tteggelhpcloud deploy complete "Normality has been restored. Anything you can't cope with is therefore your own problem." :)09:53
*** unicell1 has joined #openstack-infra09:55
*** basha has joined #openstack-infra09:56
*** unicell has quit IRC09:58
*** gokrokve has joined #openstack-infra10:02
*** guangjun has quit IRC10:02
*** Longgeek has quit IRC10:03
*** Longgeek has joined #openstack-infra10:03
*** gokrokve has quit IRC10:07
*** skolekonov has joined #openstack-infra10:07
*** Longgeek has quit IRC10:07
*** dims__ has joined #openstack-infra10:08
*** unicell1 has quit IRC10:11
*** hashar has quit IRC10:11
*** hashar has joined #openstack-infra10:12
*** jcoufal has joined #openstack-infra10:13
*** dims__ has quit IRC10:13
*** jgallard has quit IRC10:14
*** Hal_ has quit IRC10:15
*** Longgeek has joined #openstack-infra10:21
*** lcheng has quit IRC10:22
*** NikitaKo_ has quit IRC10:24
*** lcheng has joined #openstack-infra10:27
*** matrohon has quit IRC10:32
*** dims__ has joined #openstack-infra10:32
*** roeyc has joined #openstack-infra10:38
*** fabo has quit IRC10:41
*** fabo has joined #openstack-infra10:41
openstackgerritRoman Podoliaka proposed a change to openstack-infra/config: Test oslo.db against all supported SA branches
*** matrohon has joined #openstack-infra10:45
sdaguederekh_: I don't think any of the grok rules actually match your logs10:46
*** mdenny has quit IRC10:49
sdaguewhere is your log format definition for these logs in the logstash indexer?10:49
*** mdenny has joined #openstack-infra10:49
openstackgerritA change was merged to openstack-infra/elastic-recheck: Restrict 1336172 query on build_name
*** ildikov_ has joined #openstack-infra10:57
*** andreykurilin_ has joined #openstack-infra10:57
derekh_sdague: I didn't know there was such a thing, so I guess they need to be added to agent.conf ?10:59
*** asselin has quit IRC10:59
*** asselin has joined #openstack-infra10:59
*** ildikov has quit IRC11:00
sdagueok, how did you expect this to work? :) Maybe we can work backwards from there and figure out what's missing in the config.11:02
*** gokrokve has joined #openstack-infra11:02
*** e0ne has quit IRC11:03
*** maxbit has joined #openstack-infra11:03
derekh_sdague: I thought that I told logstash the file to index and then it just appeared, anyways will push an update to the config11:05
derekh_once I figure it out11:05
sdagueso you need to tell it the kind of file it is, as we used that for indexing.11:06
sdagueI think the issue is you need to add syslog to the tags11:07
sdaguebecause these look like syslog files11:07
derekh_sdague: cool, will have an update in a bit, once I'm back, thanks11:07
*** gokrokve has quit IRC11:08
*** Longgeek has quit IRC11:10
*** Longgeek has joined #openstack-infra11:10
*** andreykurilin_ has quit IRC11:12
*** otherwiseguy has joined #openstack-infra11:15
*** yaguang has quit IRC11:16
*** otherwiseguy has quit IRC11:19
*** hashar has quit IRC11:23
ttxnotmyname: for swift 2.0.0-rc2 I think the right way to do it at this point would be to cut a temporary proposed/2.0.0 branch and backport to that.11:26
ttxwe might need to merge first though (support for proposed/* ACLs)11:27
ttxfungi: when you're up, consider re-approving ^ and modifying https://$GERRIT/#/admin/projects/All-Projects,access too11:29
*** yaguang has joined #openstack-infra11:30
*** tkelsey has quit IRC11:32
*** e0ne has joined #openstack-infra11:33
*** bcrochet|afk is now known as bcrochet11:33
*** mkoderer has joined #openstack-infra11:40
*** julim has joined #openstack-infra11:45
*** ArxCruz has joined #openstack-infra11:47
*** amitpp has quit IRC11:50
*** NikitaKo_ has joined #openstack-infra11:54
openstackgerritKashyap Chamarthy proposed a change to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755
uvirtbotLaunchpad bug 1336755 in ceilometer " test times out waiting for Nova notification" [Undecided,New]
*** saper has joined #openstack-infra11:55
*** NikitaKo_ has quit IRC11:55
*** NikitaKo_ has joined #openstack-infra11:56
openstackgerritA change was merged to openstack-infra/config: Change ACLs to apply on proposed/* branches
*** nosnos has quit IRC11:59
*** nosnos has joined #openstack-infra11:59
*** Shrews has quit IRC12:00
*** yaguang has quit IRC12:01
*** rlandy has joined #openstack-infra12:02
*** Shrews has joined #openstack-infra12:02
*** gokrokve has joined #openstack-infra12:02
*** Longgeek_ has joined #openstack-infra12:02
*** mbacchi has joined #openstack-infra12:03
*** nosnos has quit IRC12:04
*** jgallard has joined #openstack-infra12:04
*** Longgeek has quit IRC12:06
openstackgerritJamie Lennox proposed a change to openstack/requirements: Add requests-mock package to global requirements
*** unicell has joined #openstack-infra12:06
*** basha has quit IRC12:07
*** gokrokve has quit IRC12:07
*** Longgeek_ has quit IRC12:07
*** Longgeek has joined #openstack-infra12:08
*** mugsie has joined #openstack-infra12:08
*** otherwiseguy has joined #openstack-infra12:14
*** annegent_ has joined #openstack-infra12:15
*** alexpilotti has joined #openstack-infra12:15
*** rfolco has joined #openstack-infra12:16
*** yaguang has joined #openstack-infra12:18
*** otherwiseguy has quit IRC12:18
*** doug-fish has joined #openstack-infra12:19
*** annegent_ is now known as annegentle_12:21
*** mdenny has quit IRC12:22
openstackgerritCyril Roelandt proposed a change to openstack/requirements: Bump glanceclient to 0.13.1
*** mdenny has joined #openstack-infra12:22
openstackgerritA change was merged to openstack/requirements: Add python-marconiclient
*** doude has quit IRC12:25
openstackgerritA change was merged to openstack/requirements: Update python-cinderclient>=1.0.7
*** pdmars has joined #openstack-infra12:27
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Update openstack-doc-tools
*** dstanek_zzz is now known as dstanek12:29
*** doude has joined #openstack-infra12:30
*** pdmars_ has joined #openstack-infra12:31
*** pdmars has quit IRC12:31
openstackgerritA change was merged to openstack-infra/release-tools: Add bug-deferral feature to
openstackgerritA change was merged to openstack-infra/release-tools: Support proposed/* branches in release scripts
*** Hal_ has joined #openstack-infra12:34
*** mestery has quit IRC12:35
*** hashar has joined #openstack-infra12:37
*** talluri has quit IRC12:38
*** talluri has joined #openstack-infra12:38
*** mestery has joined #openstack-infra12:39
*** otherwiseguy has joined #openstack-infra12:39
*** unicell has quit IRC12:39
openstackgerritA change was merged to openstack-infra/release-tools: Support swift and oslo milestone releases
*** BobBall_ has joined #openstack-infra12:42
openstackgerritA change was merged to openstack-infra/release-tools: The argparse module was added in requirements
*** BobBall has quit IRC12:42
*** talluri has quit IRC12:43
*** dprince has joined #openstack-infra12:44
rgerganovsdague, hi, do you have credentials for I'd like to make a one line change in hideci.js12:45
*** adalbas has joined #openstack-infra12:45
sdaguergerganov: no, sorry.12:45
sdagueSergeyLukjanov or fungi probably do12:46
rgerganovsorry for bothering you guys, but there is no easy way to test this locally12:46
*** BobBall_ has quit IRC12:47
*** BobBall has joined #openstack-infra12:47
*** BobBall is now known as Guest6092112:47
*** unicell has joined #openstack-infra12:50
SergeyLukjanovrgerganov, hey12:50
SergeyLukjanovrgerganov, which kind of access do you need?12:51
rgerganovSergeyLukjanov, I'd like to change ciRegex in
*** baohua has quit IRC12:52
rgerganovSergeyLukjanov, it should be / Dev$/12:52
*** baohua has joined #openstack-infra12:52
rgerganovthat is a hack to test my pending patch for hiding CI comments12:52
*** dims__ has quit IRC12:52
*** julim has quit IRC12:53
SergeyLukjanovstoring my tab name here ;)12:53
*** dims__ has joined #openstack-infra12:53
SergeyLukjanovrgerganov, it should be updated on the host?12:54
rgerganovSergeyLukjanov, yes12:54
*** weshay has joined #openstack-infra12:54
*** julim has joined #openstack-infra12:54
*** radez_g0n3 is now known as radez12:55
fungilooks like /home/gerrit2/review_site/static/hideci.js on review-dev was last updated three days ago12:55
SergeyLukjanovfungi, morning12:55
*** annegentle_ has quit IRC12:55
SergeyLukjanovfungi, I'm going to update it now, is it ok?12:55
rgerganovfungi, yeah, mordred deployed it12:55
fungiSergeyLukjanov: sure, go ahead. i was hoping we had a small puppet change to deploy it just to review-dev, but apparently not12:57
fungilooks like it was just done manually12:57
SergeyLukjanovfungi, yeah, I've not find it too12:57
*** jlibosva has quit IRC12:57
SergeyLukjanovrgerganov, done12:58
*** dkranz has joined #openstack-infra12:58
rgerganovSergeyLukjanov, thanks!12:58
fungiSergeyLukjanov: looks like it may also require a gerrit restart there (at least i see one in the shell history performed immediately following mordred's edit)12:59
*** jlibosva has joined #openstack-infra12:59
*** pblaho has quit IRC12:59
rgerganovhm, the rest of the changes are gone12:59
SergeyLukjanovfungi, the js script updated already12:59
rgerganovwhat about GerritSiteHeader.html? I think it is reverted13:00
rgerganovthe "Toggle CI" button doesn't show up13:00
*** BobBall_ has joined #openstack-infra13:00
kashyapHeya folks, Sean Dague suggested an idea on #openstack-qa to host the Fedora image (for DevStack download) to since there have been mirroring issues from external mirrors13:01
*** mdenny has quit IRC13:01
*** BobBall_ is now known as GnomeKing13:01
*** dangers_away is now known as dangers13:02
*** mfer has joined #openstack-infra13:02
*** gokrokve has joined #openstack-infra13:02
rgerganovSergeyLukjanov, fungi now my browser requests which result in 40413:02
rgerganovI have no idea where this URL parameter comes from13:03
*** chandan_kumar is now known as chandankumar13:03
*** Guest60921 has quit IRC13:03
fungirgerganov: is it possible the other files are managed by puppet? if so, it probably overwrote them back to their previous contents configured in the openstack-infra/config project13:05
*** tkelsey has joined #openstack-infra13:05
kashyapI wonder what do admins here think for the above suggestion13:05
rgerganovfungi, this is the puppet change in my patch:
*** yaguang has quit IRC13:06
fungikashyap: that seems like something we'd only want to turn on in our automated test runs, and not in general for all users of devstack. also having some mechanism to update that cached copy when upstream's image gets changed would probably be warranted13:06
kashyapafazekas, sdague ^13:06
*** gokrokve has quit IRC13:07
kashyapfungi, Why not for all users of devstack? Just asking out of ignorance13:07
fungikashyap: we already cache it onto our devstack snapshot images using nodepool, anyway, so it shouldn't be downloaded on every run, but only when we're creating new images13:07
fungikashyap: we're not the fedora project, so ideally we shouldn't expect devstack users to get fedora images from us13:08
fungikashyap: just like we don't intend for devstack users to get pypi packages from our pypi mirror, but rather from the real pypi13:08
kashyapfungi, True, this is not a long term solution,13:08
*** MarkAtwood has joined #openstack-infra13:08
kashyapI'm only trying to see if the mirroring issues are for real from Fedora Project13:08
kashyapThe admins (Kevin Fenzi) there frequently ask me, if OpenStack upstream faces mirroring issues, and have specific details13:09
kashyapThey're ready to jump on it to help out13:09
fungikashyap: so to take a step back, what problem are you trying to solve? have we been having stale images due to nodepool image update failures from fedora image download problems?13:09
fungior something else?13:09
kashyapfungi, Let me see if I can summarize -13:10
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Search use case 1: Quicknav
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: fix fingerprint to deal with n-obj hit
fungikashyap: because we don't download fedora images for every devstack job we run. we bundle it into the snapshots which we use to spin up devstack test nodes, and that only gets refreshed daily13:10
sdaguefungi: right, but if we have a 10% failure rate pulling into nodepool, then how does that impact nodepool builds?13:11
kashyapfungi, Fair enough (I understand the case where it's snapshotted & reused)13:11
*** xianghui has quit IRC13:12
fungisdague: 1 in 10 nodepool images end up being a day older than the others13:12
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: DB api for teams
*** e0ne has quit IRC13:12
* fungi checks to see if it's happening now13:12
sdagueyeh, part of what's hard here is the nodepool logs not being available outside of infra-root13:12
*** MarkAtwood has quit IRC13:13
kashyapAh, that was what I was about to ask, if the logs are available13:13
fungisdague: the nodepool image update logs *are* published and readily excessible to everyone13:13
*** e0ne has joined #openstack-infra13:13
*** MarkAtwood has joined #openstack-infra13:14
*** GnomeKin1 has joined #openstack-infra13:14
derekh_If this file out of date I can't find anywhere, I'm guessing I should be looking at log-gearman-worker.py13:14
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Added custom CSS aliases for common storyboard icons.
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Auth error handling
sdaguefungi: oh,13:15
*** eharney has joined #openstack-infra13:15
fungianyway, a quick survey of our 8 locations where we build devstack-trusty snapshots seem to have successfully updated twice in the past 48 hours... i'll check again around 15:00 when today's runs complete13:15
rgerganovSergeyLukjanov, do you have a minute to check something on the review-dev host?13:15
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Notification Priority Constant
fungiderekh_: maybe what you're looking for is in
*** GnomeKing has quit IRC13:17
sdaguefungi: well I don't see errors in there, but the nodepool logs are pretty interleaved, so it's a little hard to see what's going on :)13:18
derekh_fungi: cool, thanks13:18
fungiderekh_: i think it got split out into its own top-level puppet module on a refactor13:18
fungisdague: yeah, it's suboptimal, but the data is there. i end up filtering for the provider+label i'm interested in13:19
fungisdague: but i'm sure we wouldn't turn down improvements to nodepool's logging which make it easier for others to diagnose failures13:19
sdaguefungi: ok, I stick it on my queue.... so sometime around the Q release :)13:20
*** MarkAtwood has quit IRC13:20
*** dkliban_afk is now known as dkliban13:20
*** yaguang has joined #openstack-infra13:21
*** mriedem has joined #openstack-infra13:22
fungisdague: so soon?!?13:22
*** NikitaKo_ has quit IRC13:23
openstackgerritA change was merged to openstack-infra/elastic-recheck: fix fingerprint to deal with n-obj hit
fungianyway, i don't currently see image updates impacted by cache refresh download issues, but if/when it does happen the impact is negligible since we just keep using the previous day's image wherever we failed to update13:24
fungiand we could always stick in some additional retry logic if it does become inconvenient for some reason13:24
*** GnomeKin1 is now known as Bobba13:24
fungialso, if we start building one image and uploading it to all providers via glance, the number of times we download it would be the same whether we're building a new image or updating our network mirror13:25
*** rgerganov has quit IRC13:25
sdaguewell, is that true?13:26
sdaguebecause we'd pull it to the mirror once, it would be known good13:26
sdagueor are you seeing the image build with yesterday's image?13:26
sdaguevs. building from a clean base13:26
fungiwould we only pull it to the mirror once? does fedora not release new images?13:26
fungibut also i believe dib does its own local caching, so wouldn't redownload files unnecessarily (others who have worked directly with it should correct me if i've misunderstood)13:27
sdaguefungi: these images are very specifically version tagged13:28
*** xianghui has joined #openstack-infra13:29
fungiand we don't update the reference to them in devstack?13:29
sdaguenot very often13:29
sdaguethis is the image that's needed for heat testing13:30
fungii suspect we'd still need to dynamically identify when new versions are in need of being downloaded, whether the trigger is new versions appearing on a remote site or new version references in devstack itself, and automatically act accordingly13:30
sdagueafazekas has the 3rd update for that image in 8 months pending13:30
*** annegent_ has joined #openstack-infra13:31
fungicurrently nodepool figures out what to download by inspecting devstack source i think, but locally caching that and reusing it across rebuilds if it's still the same shouldn't be too much effort13:31
*** julim has quit IRC13:34
*** prad has joined #openstack-infra13:35
*** julim has joined #openstack-infra13:35
sdaguefungi: actually we have an image_list.sh13:35
sdaguebecause the source inspection was really error prone13:36
fungiyeah, that. i remember when it was added now13:36
*** lcheng has quit IRC13:37
*** pcrews has joined #openstack-infra13:37
*** jlibosva has quit IRC13:38
*** lcheng has joined #openstack-infra13:38
*** jlibosva has joined #openstack-infra13:39
fungihrm, so looking at, since it runs on a fresh template we'd need to push the pre-cached files onto it if we wanted to reuse them, so we'd likely need some separate mechanism on the nodepool server to collect, cache and dump them to the templates13:40
*** sgordon has joined #openstack-infra13:40
sgordonanyone looked at ask.o.o today? seems we have a bit of a spam issue...13:40
kashyapSorry fungi - My internet at my parents is abysmal, and my IRC proxy went bonkers, if you addressed any message for me, likely I haven't got it13:40
fungibut with dib, as i said, i think there's caching already build in13:40
fungisgordon: that's awesome13:41
kashyapsgordon, Yeah, I looked at it, the same crap was plagued to ask.fp.o13:41
sgordonfungi, i find it amusing that you only get 5 "report abuse" points per day13:42
kashyapsgordon, I guess I even called that spammer on that number & asked him to refrain (on a request from a ask.fp.o admin)13:42
fungireed: ^ fyi13:42
sgordonfungi, but it seems you can create around 15 questions in 2 minutes13:42
sgordonbefore having to create another account (which they did)13:42
fungisgordon: yeah, amusing indeed13:42
*** annegent_ has quit IRC13:43
*** annegent_ has joined #openstack-infra13:44
*** wenlock has joined #openstack-infra13:44
*** jpeeler has quit IRC13:45
*** jpeeler has joined #openstack-infra13:45
*** jpeeler has quit IRC13:45
*** jpeeler has joined #openstack-infra13:45
*** prad has quit IRC13:47
*** homeless has joined #openstack-infra13:51
*** thedodd has joined #openstack-infra13:51
*** unicell has quit IRC13:52
*** lcheng has quit IRC13:53
*** baohua has quit IRC13:54
*** miqui has quit IRC13:56
*** amitpp has joined #openstack-infra13:57
*** NikitaKo_ has joined #openstack-infra13:58
*** mestery_ has joined #openstack-infra13:59
*** manueldesbonnet has joined #openstack-infra13:59
*** mestery has quit IRC14:00
*** mestery_ is now known as mestery14:00
*** amitpp has quit IRC14:00
*** amalagon has joined #openstack-infra14:01
*** lcheng has joined #openstack-infra14:01
*** gokrokve has joined #openstack-infra14:02
*** zz_dimtruck is now known as dimtruck14:03
*** yaguang has quit IRC14:03
*** julim has quit IRC14:03
*** julim has joined #openstack-infra14:04
dhellmannfungi: thanks for tracking down that wheel issue and getting the fix pushed through14:04
* dhellmann makes a note not to take off mondays any more14:04
* fungi makes a note to take mondays off instead14:05
fungisgordon: reed: i think we've squashed all the spambot posts for now, but should probably keep an eye out for more14:06
sgordonfungi, it would be preferable to actually delete them14:06
*** gokrokve has quit IRC14:07
sgordonfungi, for that kind of spammer the closed state with all the text still present is good enough for them14:07
*** MarkAtwood has joined #openstack-infra14:07
sgordonso not going to dissuade them in any way14:07
fungisgordon: agreed. hopefully reed will get in touch with evgeny and get that to happen at some point14:07
fungisgordon: reporting a bug about it at may also help14:08
*** MarkAtwood has quit IRC14:08
fungisgordon: note that the openstack infrastructure team has no access to that system, so can't really do much about it14:08
sgordondoes anybody then?14:09
fungisgordon: might be that there are ask admins who can delete posts through the webui, though i don't know who they are... hopefully they have some bulk delete/hide feature at their disposal14:10
*** rwsu has joined #openstack-infra14:10
*** wenlock has quit IRC14:10
*** jlibosva has quit IRC14:11
fungihmm... looks like ask calls them "moderators"14:11
*** jlibosva has joined #openstack-infra14:11
fungi implies moderators can delete questions14:13
manueldesbonnetHowdy, I'm looking to get approved, if possible (it's a change to pypimirror to allow specifying requirements files on cmdline).14:13
manueldesbonnetAny takers?14:13
*** hashar has quit IRC14:14
*** salv-orlando has joined #openstack-infra14:14
fungisgordon: oho, looks like i actually have sufficient karma to be a moderator, but someone else has already deleted them now anyway14:15
fungisgordon: i guess worth keeping in mind for next time14:16
sgordoni dont suppose you know what the magic number is14:16
sgordoni am still on ~140014:16
fungisgordon: the top-right corner of the main interface claims 10014:16
*** yaguang has joined #openstack-infra14:17
*** vhoward- has joined #openstack-infra14:17
fungisgordon: when i view a question i see a somewhat discreet "delete" trashcan next to the "close" button14:17
sgordonah yes14:18
*** vhoward- has left #openstack-infra14:18
*** e0ne has quit IRC14:19
*** basha has joined #openstack-infra14:19
*** msabramo has quit IRC14:21
*** afazekas has quit IRC14:21
*** yfried has joined #openstack-infra14:22
*** NikitaKo_ has quit IRC14:25
*** gokrokve has joined #openstack-infra14:26
*** Hal_ has quit IRC14:27
*** gokrokve_ has joined #openstack-infra14:27
*** Hal_ has joined #openstack-infra14:30
yolandajeblair, working on changes for nodepool. Looking at your review, looks as if you want to drop the old method of uploading images using snapshots?14:30
*** nelsnelson has quit IRC14:30
*** nelsnelson has joined #openstack-infra14:30
*** gokrokve has quit IRC14:31
*** zz_gondoi is now known as gondoi14:31
anteayaI had a cranky router this morning14:32
*** wenlock has joined #openstack-infra14:32
anteayaArxCruz: thanks for letting me know14:33
ArxCruzanteaya: np :)14:33
ArxCruzstill working, i'll let you know when it's done :)14:33
*** msabramo has joined #openstack-infra14:33
*** sabeen2 has joined #openstack-infra14:34
anteayakrtaylor: yeah, nothing listed though I'm confident the topic will come up in many ways, not the least of which when we discuss vinz and gerrit gui14:34
anteayaArxCruz: kk, thanks14:34
*** jcoufal has quit IRC14:34
anteayaArxCruz: is your system disabled then?14:34
*** prad has joined #openstack-infra14:34
ArxCruzanteaya: no, but it's not voting, I need to have enabled to test :/14:35
anteayawell as long as it isn't voting14:35
anteayacan you have it enabled on sandbox only and have it test there?14:35
anteayathe community doesn't like to be the testing platform for things14:36
anteayaunless you have conscious consent14:36
anteayabecause when you do that14:36
anteayayou train people to ignore your system14:36
*** jcoufal has joined #openstack-infra14:36
anteayawhich is a bad choice14:37
*** sabeen2 has quit IRC14:37
*** sabeen1 has quit IRC14:38
*** sabeen2 has joined #openstack-infra14:38
krtayloranteaya, I was asking because there was a small chance I could go to mid-cycle, hard to justify w/o CI content14:38
anteayaah okay14:39
krtayloranteaya, I'll still offer to have a mid-cycle for 3rd party systems in Austin :)14:39
anteayalet me help you justify14:39
anteayaright, a bit late to plan that one I think14:39
anteayabut perhaps we can plan if for k release14:39
*** signed8bit has joined #openstack-infra14:40
krtayloranteaya, and a bit early to have real content, yes perhaps for K14:40
anteayaalso the reason we don't explictly state third party business on the qa/infra meetup agenda is we tend to attract people who want tutorials on how to set up their system rather than wanting to help infra and qa work on infra and qa issues14:40
*** Bobba is now known as BobBall14:40
anteayaso I'm willing to write a letter to whomever needs to read the letter to get you to attend germany14:41
anteayasince I want you to be there14:41
anteayaso what do I do to make that happen?14:41
*** basha has quit IRC14:41
*** ociuhandu has quit IRC14:42
krtayloranteaya, understood, that is a problem14:42
krtayloranteaya, I don't think a letter is necessary, but you could help me come up with CI discussion justificsation14:43
anteayayes, and we are trying to make all the people happy, which is hard14:43
krtaylorand justification14:43
*** gokrokve_ has quit IRC14:43
anteayatoss some words on an etherpad and we can move them around until you have what you need14:43
anteayahow's that14:43
krtayloryes, hard, but we could get a lot of doc/definitions work done14:44
krtayloranteaya,  will do14:44
anteayahey I could put infra-manual on the agenda14:44
anteayawould that help?14:44
anteayaI plan on working on that anyway14:44
*** jgrimm has joined #openstack-infra14:44
tteggelhpcloud instance boot times high at the moment.. we're working the issue14:44
*** otherwiseguy has quit IRC14:44
anteayatteggel: thanks for the heads up14:44
anteayaappreciate it14:44
krtayloranteaya,  do we have a useful set of CI folks attending?14:45
tteggelanteaya: you're welcome14:45
* krtaylor needs to review attending14:45
anteayatteggel: anything we can do to help?14:45
*** yaguang has quit IRC14:45
tteggelanteaya: don't think so. our DB is in meltdown. we have an couple of mysql experts in attendance14:45
anteayakrtaylor: well all of infra is involved with helping third party be as smooth as possible14:45
anteayaI'm just playing point14:46
anteayatteggel: very good, don't let me interupt14:46
anteayatteggel: do say if we can help with anything14:46
tteggelanteaya: will do14:46
anteayatteggel: thanks14:46
anteayakrtaylor: and we will be discussing vinz, which is the hopeful gerrit gui replacement14:47
*** gokrokve has joined #openstack-infra14:47
anteayaso the whole ml discussion about how to represent third party ci will get much air time14:47
anteayaand your input would be very welcome in that discusssion14:47
anteayakrtaylor: what do you need to see to decide if you want to attend?14:48
anteayabecause I guarentee there will be value for you there14:48
fungisdague: your nova change is hitting an interesting py26 unit test failure in the gate14:48
*** ArxCruz has quit IRC14:49
*** ArxCruz_ has joined #openstack-infra14:49
*** ArxCruz_ has quit IRC14:50
*** andreaf has quit IRC14:51
*** devkulkarni has joined #openstack-infra14:51
*** ArxCruz has joined #openstack-infra14:51
krtayloranteaya, its not me, its getting approval, need strong case14:52
*** mfer has quit IRC14:52
krtayloranteaya, I have to drive to office, bbiab and we'll discuss more14:52
sdaguefungi: yeh14:53
sdaguehonestly, I've not landed an openstack without 3 rechecks in a week :)14:53
sdagueand they are all different fails14:53
sdagueso my energy to dig has been exhausted14:54
*** tkelsey has quit IRC14:54
mtreinishsdague: push the patch to remove py26 testing from nova14:54
mtreinishit seems like a justified response :)14:54
*** mfer has joined #openstack-infra14:55
*** dims__ has quit IRC14:55
*** ociuhandu has joined #openstack-infra14:55
anteayakrtaylor: kk14:56
anteayadrive safe14:56
*** vhoward- has joined #openstack-infra14:56
*** yaguang has joined #openstack-infra14:58
*** miqui has joined #openstack-infra14:59
*** e0ne has joined #openstack-infra15:02
*** krtaylor has quit IRC15:05
*** jaypipes-GOOAL has quit IRC15:06
*** tkelsey_ has joined #openstack-infra15:07
*** doug-fish has quit IRC15:08
*** doug-fish has joined #openstack-infra15:09
*** gokrokve has quit IRC15:10
*** mrmartin has quit IRC15:11
*** gokrokve has joined #openstack-infra15:11
*** virmitio has joined #openstack-infra15:13
*** moted has joined #openstack-infra15:13
*** MarkAtwood has joined #openstack-infra15:17
*** doug-fish has quit IRC15:17
*** doug-fish has joined #openstack-infra15:18
*** xianghui has quit IRC15:18
*** matrohon has quit IRC15:18
*** packet has joined #openstack-infra15:19
*** hashar has joined #openstack-infra15:23
*** dims__ has joined #openstack-infra15:23
mordredmtreinish: :)15:24
*** jlibosva has quit IRC15:25
*** gokrokve has quit IRC15:25
*** dims__ has quit IRC15:27
*** dims__ has joined #openstack-infra15:27
*** matrohon has joined #openstack-infra15:28
*** gondoi is now known as zz_gondoi15:32
*** zz_gondoi is now known as gondoi15:35
*** unicell has joined #openstack-infra15:36
*** james_li has joined #openstack-infra15:36
*** mzoeller has quit IRC15:36
*** deny_ has joined #openstack-infra15:39
*** yfried has quit IRC15:40
openstackgerritA change was merged to openstack/requirements: Update glanceclient to 0.13.1
*** misterpink has quit IRC15:40
hashargood morning!15:40
hasharif you ever heard of Gerrit hash mismatch bug of doom.  We apparently got the solution
hasharApache Mina SSHD needs to be updated.   pelix told me he was going to try a new version of Gerrit that got that bugfix15:41
zarojhesketh, clarkb : i believe nodepool is on jenkins-dev
*** jcoufal has quit IRC15:42
*** otherwiseguy has joined #openstack-infra15:42
*** yfried has joined #openstack-infra15:43
*** yaguang has quit IRC15:44
*** afazekas has joined #openstack-infra15:44
*** msabramo has quit IRC15:44
*** chandankumar has quit IRC15:44
zarohashar: good to know.  which version of gerrit was that bug introduced?15:44
hasharzaro: apparently related to Java version being used15:46
hasharzaro: our bug report is quite old. I don't think anyone bothered to bisect the multiple softwares involved15:46
*** changbl has quit IRC15:47
*** gangil has joined #openstack-infra15:48
openstackgerritJulien Vey proposed a change to openstack-infra/config: Add solum-infra-guestagent repo to stackforge
*** nati_ueno has joined #openstack-infra15:50
*** ihrachyshka has quit IRC15:51
zarohashar: we are on java '(IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~' but have not run into the problem yet, or not that anyone has reported.15:53
hasharI have no idea which version of java we are running :/15:53
*** pdmars_ has quit IRC15:54
openstackgerritEmilien Macchi proposed a change to openstack-infra/config: Move puppet-trove to Stackforge
fungizaro: hashar: actually we run into it all the time in our git-review functional testing15:55
*** pballand has joined #openstack-infra15:55
*** msabramo has joined #openstack-infra15:55
jeblairclarkb, jhesketh: i've put jenkins07 in shutdown mode (maybe jobs will be closer to being done when clarkb comes online)15:56
fungizaro: hashar: the reason we don't see it in public is that if you have bouncy castle loaded the problem method seems to not be in the code path any longer (presumably one of the routines offloaded to bc)15:56
pelixzaro: hashar: the latest gerrit 2.8.6 has the fix, but broke other stuff (namely DB transactions), but it may still work for the unit tests15:56
*** roeyc has quit IRC15:56
openstackgerritEmilien Macchi proposed a change to openstack-infra/config: Move puppet-trove to Stackforge
tteggelhpcloud instance build time issue resolved. should be back to normal.15:57
fungithe details of the error in mina-sshd are amusing too. if the generated key material starts with a zero byte, some functions strip it causing a mismatch 1 in 256 times15:57
*** Sukhdev has joined #openstack-infra15:58
hasharthough on the gerrit-review integration test that happen more frequently apparently15:58
*** bdpayne has joined #openstack-infra15:58
fungitteggel: great--thanks for the update! luckily it didn't seem to impact us heavily (at least not that i noticed in the graphs)15:58
hasharwe have a patch that haven't yet landed after 6 rechecks or so15:58
fungihashar: i don't think it really happened more often... i think that we actually make roughly that many ssh connections over the course of one test run15:58
*** IvanBerezovskiy has left #openstack-infra15:58
*** otherwiseguy has quit IRC15:58
hasharfungi: oh of course!15:58
fungiso the chances that one gets fouled up by the 1:256 chance is extremely high15:59
*** skolekonov has quit IRC15:59
*** pballand has quit IRC16:00
ttxjeblair: ping re: using proposed/* now for swift 2.0.0-rc216:00
fungiif 2.8.6 incorporates an updated mina-sshd with the fix, i'll try a patch switching to that in the tox env16:00
*** signed8b_ has joined #openstack-infra16:00
jeblairttx: i suspect we may have some gerrit acl work to do; will check16:00
*** otherwiseguy has joined #openstack-infra16:00
ttxjeblair: context: swift 2.0.0 needs a few commits backported, RC1 was tagged on master16:01
fungijeblair: i updated the all-projects acl this morning when i reapproved ttx's rebased acl patch16:01
ttxjeblair: my proposal would be to create a temporary proposed/2.0.0 branch and backport to it, tag on it16:01
jeblairttx: i'm guessing fungi already did that :)16:01
jeblairfungi: yay16:01
*** signed8bit has quit IRC16:01
zarofungi: maybe you should try on 2.9, it's about to be released any day now.16:01
ttxjeblair: afaik we have the tooling aligned now, merge_tags was also pushed16:02
*** anode has joined #openstack-infra16:02
*** mestery has quit IRC16:02
ttxjeblair: if you're right that the magic should just happen with proposed/foo, then I think that will work16:02
fungizaro: there's a balance between testing git-review with a gerrit version people are likely to already have running in public, and a version of gerrit which doesn't cause tests to arbitrarily fail. but yes i expect 2.9 will start getting tested for git-review changes soon16:02
jeblairttx: ok, i can't think of anything that would block; i can stand by to assist if there are problems16:02
ttxcool, let me try to manually create the branch from the RC1 tag16:03
*** gyee has joined #openstack-infra16:03
zarofungi: 2.8.x doesn't have the CHANGE_OWNER acl, 2.9 does.16:03
* ttx engages manual mode16:03
*** pballand has joined #openstack-infra16:03
fungizaro: awesome--i can't wait for us to upgrade so we have access to start using that16:03
tteggelfungi: welcome. i want to see the error rate from hpcloud drop too16:04
fungizaro: did they keep the old change screen around in 2.9 after all?16:04
ttxjeblair: the idea would be to merge tags and then delete that proposed/2.0.0 branch16:04
jeblairttx: sounds good16:05
*** bdpayne has quit IRC16:05
zarofungi: hmm, i believe so. might default to new change screen but default can be set in gerrit.config16:06
*** gokrokve has joined #openstack-infra16:06
ttxjeblair: ok, proposed/2.0.0 branch craeted, post job making a tarball from it16:08
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Update tests to use Gerrit 2.8.6
*** arnaud has joined #openstack-infra16:08
*** ildikov has joined #openstack-infra16:09
jeblair2014-07-02 16:08:20.365 | [SCP] uploading file: '/srv/static/tarballs/swift/swift-proposed-2.0.0.tar.gz'16:09
zarofungi: speaking of.. would it be possible to move this along?
ttxjeblair: sounds legit16:10
zarofungi: it's been in the que for a while and i believe rcarrillocruz is waiting for it to lang.16:10
*** lcheng has quit IRC16:10
fungizaro: does that appear in 2.8.5?16:11
*** ildikov|2 has joined #openstack-infra16:11
*** ildikov has quit IRC16:11
zarofungi: no, it's not back ported.16:12
fungihrm... looks like it's not included in any tags16:12
fungiokay, so this is a backport to 2.8 of something earmarked for 2.9 or 3.0?16:12
*** ildikov_ has quit IRC16:12
mordredjeblair: neat!16:13
*** paul-- has quit IRC16:14
fungizaro: if we approve 94675 we still need to update the config repo to refer to the new warfile url, right?16:15
jeblairyesterday, i learned that sheng kee makes hong kong style pineapple buns!  i'm so excited because i've been jonesing and that was looking to be an expensive breakfast.16:15
fungizaro: also, do you know whether any of our current hook scripts will break when gerrit starts applying the new command-line option?16:16
*** otherwiseguy has quit IRC16:16
ttxjeblair: notmyname should push a few backports to that branch soon. Could you keep an eye on the tests and make sure they do what they are supposed to do ?16:16
jeblairttx, notmyname: let me know when16:17
ttxI think they will, the use case is not that weird (shall use master everywhere else)16:17
*** rgerganov has joined #openstack-infra16:17
fungihashar: i'm exercising 2.8.6 in to confirm the bug is gone... first pass was all green!16:18
*** gangil has left #openstack-infra16:18
*** paul-- has joined #openstack-infra16:19
zarofungi: yes, an update to load new ware would be required as well.16:19
reedsgordon, fungi: evgeny is working on a fix for the bot issue, ETA a couple of days... hopefully16:19
*** mestery has joined #openstack-infra16:19
fungireed: awesome--thanks for the heads up16:19
*** atiwari has joined #openstack-infra16:19
*** mriedem has quit IRC16:19
*** jcoufal has joined #openstack-infra16:20
*** doude has quit IRC16:20
pleia2good morning16:20
zarofungi: i haven't tested but should be completely backwards compatible. IIRC the scripts in jeepyb retreives the info by name.  rcarrillcruz wanted to update jeepyb with the additional info.16:22
annegent_reed: around?16:24
fungizaro: okay, great. the only hook i specifically remember breaking on unrecognized options was the trivial rebase hook, which we don't use now16:24
reedannegent: yes16:25
*** ArxCruz has quit IRC16:25
*** ArxCruz has joined #openstack-infra16:26
hasharfungi: awesome!16:26
hasharfungi: gotta move out. bbl16:26
openstackgerritJulien Vey proposed a change to openstack-infra/config: Add solum-infra-guestagent repo to stackforge
*** hashar has quit IRC16:27
*** krtaylor has joined #openstack-infra16:30
*** markmc has quit IRC16:31
zarofungi: from looking at the code, it looks like it will just ignore the new arg.
*** basha has joined #openstack-infra16:31
*** david-lyle has joined #openstack-infra16:32
*** ArxCruz has quit IRC16:33
*** ArxCruz_ has joined #openstack-infra16:33
*** ildikov|2 has quit IRC16:33
fungizaro: well, yeah, as long as our other hook scripts are written similarly then they should also be fine (remember that this option is being passed as part of all of those hook calls, so every hook script needs to be written to ignore unrecognized options)16:33
*** nati_ueno has quit IRC16:33
*** flaper87 is now known as flaper87|afk16:34
ruhemordred: hi! question about recent changes in now it uses ; who controls this mirror and how can i push a specific package to that mirror?16:35
pelixhashar: I can confirm we'll have to wait for the gerrit 2.8.7 to test git-review against release as 2.8.6 clearly has some issues...16:35
*** bdpayne has joined #openstack-infra16:35
*** basha has quit IRC16:35
*** ArxCruz_ has quit IRC16:35
*** ArxCruz has joined #openstack-infra16:36
fungiruhe: that mirror is a direct copy of pypi.python.org16:36
ruhefungi: hi. do you have any idea why a package (which is present on wouldn't be present in this mirror?16:37
ruhefungi: here is the package
openstackgerritA change was merged to stackforge/python-jenkins: Adds assert_node_exists()
fungiruhe: checking16:38
zarofungi: ohh of course. i'll double check the other ones.16:38
*** dizquierdo has quit IRC16:38
ruhefungi: thanks16:38
*** dkehn__ has joined #openstack-infra16:39
jeblairyolanda: yes -- my guess is that it would be easiest to remove the support for snapshot-based images in nodepool.  ideally, we would keep them both, but i think the concurrency requirements are different enough that it would be compilacetd16:40
*** Ajaeger has joined #openstack-infra16:40
jeblairyolanda: so if you find that you agree and it would be easier to remove it, i think that's fine.16:40
fungimordred: ruhe has found an interesting gap... exists (and has tarball links) while there is no at all16:40
AjaegerHi infra team, I have a few docu problems. Has anybody now some time for some help, please?16:41
jeblairfungi, mordred: grep yaql /var/log/bandersnatch/mirror.log  produces nothing16:41
*** dkehnx has quit IRC16:41
yolandajeblair, ok, at the moment i was separating cases, but it will be much simpler to just drop it16:42
fungijeblair: yep16:42
yolandai can upload something for tomorrow16:42
mordredjeblair: problem with the cron job perhaps? you can run "bandersnatch mirror" at any time and it should do a mirror run real quick16:42
jeblairmordred: it's running regularly16:43
mordredk. well, that's both good and bad16:43
jeblairmordred: there's the latest output16:43
mordredoh- I mis-read the above - I thought you were saying there was no output in the log :)16:43
jeblairalso, do whe know why it's having that problem with skll every time?16:43
jeblair(it repats that output for every run)16:43
*** marcoemorais has joined #openstack-infra16:43
reedsgordon, jpich, fungi: looks better now, thanks those who helped deleting spam and blocking users16:44
jeblairfungi: i'm assuming i should have cronspam with previous mirror output, yeah?16:44
fungijeblair: yes16:44
jeblairfungi: i can't find cronspam with yaql in it16:44
*** thedodd has quit IRC16:44
jeblairso there was probably never an error reported, or if there was, it must have been in an inital run or something16:45
*** annegent_ has quit IRC16:45
*** marcoemorais has quit IRC16:45
*** ArxCruz_ has joined #openstack-infra16:45
*** marcoemorais has joined #openstack-infra16:45
*** hemna__ is now known as hemna16:45
fungiagreed. i'm digging through the fs tree currently to see if it's somehow being misplaced or under different capitalization16:46
jeblairAjaeger: how can i help?16:46
*** ArxCruz has quit IRC16:46
*** otherwiseguy has joined #openstack-infra16:46
AjaegerFirst issue is: was approved - but I do not see the docs-specs repository16:46
Ajaegerjeblair: thanks ^16:46
*** jpich has quit IRC16:47
Ajaegerjeblair: are some manual steps necessary?16:47
openstackgerritDoug Hellmann proposed a change to openstack-infra/infra-specs: Add spec for adding cross-project unit test jobs
jeblairAjaeger: no, but i think someone broke puppet yesterday16:47
Ajaegerjeblair: that would explain my second issue as well - was approved but jenkins still has the old slave script16:48
*** marcoemorais1 has joined #openstack-infra16:48
dhellmannjeblair: is it safe for me to proceed with implementing ^^? There hasn't been any feedback on the spec, yet, since I think I filed it when you guys were dealing with outages.16:49
dhellmannjeblair: well, not quite true, lifeless did leave some questions which I've answered16:49
Ajaegerjeblair: and perhaps my 3rd one as well - does not have the new 0.16.1 release...16:49
fungijeblair: mordred: nothing at all for 'find /srv/static/mirror/ | grep -i yaql' so presumably bandersnatch is just ignoring it outright16:50
*** marcoemorais has quit IRC16:50
*** zhiyan is now known as zhiyan_16:50
*** zhiyan_ is now known as zhiyan16:51
Ajaegerfungi, see also my issue above - normally pypi.o.o has a new tag within an hour but I'm waiting now for four hours for 0.16.116:51
jeblairapparently no one reviewed the emergency revert of the patcch that broke apache16:51
rgerganovmordred: could you please redeploy on I have fixed a problem with ciRegex - it was too slow16:52
*** pelix has quit IRC16:52
*** harlowja_away is now known as harlowja16:52
*** tkelsey_ has quit IRC16:52
jeblairAjaeger: i've approved the patch that should fix puppet runs, things should happen in a few hours16:53
zarofungi, rcarrillcruz:: double checked, all those jeepyb scripts use the same pattern.  will just ignore new params.16:53
jeblairAjaeger: i don't know why that would not have the latest release16:54
jeblairAjaeger: we'll need to check the post job16:54
Ajaegerjeblair: ok, will check back for the first two issues later/tomorrow. Thanks!16:54
jeblairAjaeger: i'm looking up the post job for the third one16:55
fungi seems to have uploaded16:55
Ajaegerfungi: you're too fast for me ;)16:55
jeblairAjaeger: oh, it's on
Ajaegerjeblair, fungi: It's available indeed at
fungi has it16:56
jeblairit's just not in the mirror16:56
Ajaegerjeblair: yeah, it's not in the mirrors16:57
Ajaegerand what's even strangeer is that 0.16 is in the mirror but building uses 0.1516:57
openstackgerritA change was merged to openstack-infra/config: Revert "Revert "Downgrade puppetlabs-apache to version 0.0.4.""
jeblairso this could be another manifestation of the bandersnatch problem fungi and hopefully mordred are working on16:57
*** dkehn__ is now known as dkehnx16:57
jeblairthat change ^^ should get puppet running again, so maybe in 10 mins or so we'll have new projects16:57
fungiyeo, not at
mordredjeblair, fungi: dstufft said something about bandersnatch behavior when it sees a stale thing16:57
mordreddstufft: you aroudn? what was the remediation for that again?16:58
Ajaeger - shows that 0.15 is ued16:58
*** whoops has joined #openstack-infra16:58
Ajaegerjeblair: if you think I should come check back later, no problem...16:58
jeblairmordred: i think we have 3 problems: 1) yaql is not in the mirror at all, 2) openstack-doc-tools has an old version in the mirror, and 3) the mirror emits error messages about skll:
fungiAjaeger: yep, it seems that it ended up on upstream pypi but out mirror routine hasn't pulled a copy from there16:59
jeblairdstufft: ^ see 3 problems16:59
*** jgallard has quit IRC16:59
mordredjeblair: I'm operating under the hunch that they're all related - based on something dstufft said a while back that I may have mis-interpreted17:00
mordreddstufft: namely something about noticing when a package has stale entries in the CDN17:00
Ajaegerfungi, jeblair, mordred : Thanks for looking at this! I'll keep silent watching you debug it ;)17:01
mordredI was thinking it meant that there was a possibility that we'd end up with a package stale for a little bit - but it seems that maybe it means that bandersnatch goes bonkers and there is a deeper (and crappier) problem we have to fix17:01
*** jamielennox is now known as jamielennox|away17:02
mordredjeblair: but I think dstufft can clarify and help know where we should look17:02
*** Longgeek has quit IRC17:02
fungimordred: does bandersnatch have a debug option, so we could tell it for example to update its mirror for just that package and then inspect its decision making?17:02
*** otherwiseguy has quit IRC17:03
fungithe manpage on static.o.o for bandersnatch is... let's just say unenlightening. i'll go hunting actual documentation17:04
jeblairdhellmann: spec lgtm17:04
dhellmannjeblair: cool, thanks17:04
*** otherwiseguy has joined #openstack-infra17:05
mordredfungi: it has something to do with pypi serial numbers17:05
jeblairdhellmann: for the record, i don't think i ignored that the whole time; i think i read it and saw lifeless's suggestions and thought they were good and waited for a response.  so i've only been ignoring it for a a day or so! :)17:05
dhellmannjeblair: yeah, that's sort of what I figured happened :-)17:06
*** mmaglana has joined #openstack-infra17:06
jeblairdhellmann: i'm really glad we have all those options documented.  i'm sure we're going to be reading them in 6 months :)17:07
ruhefungi: jeblair: i've just compared contents of and . the latter one is missing 1081 packages (45588 compared to 44507)17:07
dhellmannjeblair: it's like you can see into the future17:07
jeblairdstufft, mordred: so we're up to 1081 problems! :)17:07
fungimordred: yeah, i'm guessing this is related to ?17:07
*** gondoi is now known as zz_gondoi17:08
*** zhiyan is now known as zhiyan_17:08
*** SumitNaiksatam has quit IRC17:09
*** ArxCruz has joined #openstack-infra17:09
*** derekh_ has quit IRC17:11
*** zz_gondoi is now known as gondoi17:11
*** ArxCruz_ has quit IRC17:11
openstackgerritKhai Do proposed a change to stackforge/python-jenkins: Add a gitreview file
*** praneshp has joined #openstack-infra17:14
fungimordred: i'm not finding much at all in the way of debugging options for bandersnatch (unless we want to run it under pdb or something)17:15
*** johnthetubaguy is now known as zz_johnthetubagu17:15
ruhefungi: as a temporary fix (in case if this problem will not be resolved soon), would it make sense to add "extra-index-url =" to the
*** devkulkarni has quit IRC17:16
*** gondoi is now known as zz_gondoi17:16
Ajaegerjeblair: docs-specs exists - so the puppet patch fixed it!17:16
*** Ryan_Lane has joined #openstack-infra17:16
*** melwitt has joined #openstack-infra17:18
Ajaegerjeblair: still an empty repo, so puppet is not finished with doing its magic...17:19
*** dkehn__ has joined #openstack-infra17:21
*** otherwiseguy has quit IRC17:21
*** amcrn has joined #openstack-infra17:23
*** dkehn__ has quit IRC17:23
*** dkehn__ has joined #openstack-infra17:24
*** SumitNaiksatam has joined #openstack-infra17:24
*** dkehn_ has quit IRC17:24
*** dkehnx has quit IRC17:25
*** crc32 has joined #openstack-infra17:25
*** mriedem has joined #openstack-infra17:25
*** dkehn__ is now known as dkehnx17:26
*** dkehn has joined #openstack-infra17:26
*** sparkycollier has joined #openstack-infra17:29
jeblairAjaeger: there's a further puppet problem that prevented manage-projects from running on review.o.o; looking17:31
*** mbacchi has quit IRC17:33
mgagneDoes it bother anyone that program names aren't consistent across programs? "Queue service", "Block Storage", "DNS Services", etc.17:33
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Revert "Fixing deprecation warnings"
jeblairnibalizer: ^17:35
fungiruhe: hopefully we can just get a workaround in place to grab the things the mirror software is missing for whatever reason17:35
clarkbtoday is not my day17:35
clarkbjeblair: thank you for starting 0717:35
Ajaegermgagne: The documentation team had already some fun with the capitalization of Service ;)17:35
*** mwagner_ has quit IRC17:35
fungimgagne: i used to be that ocd, but then i started treating it with regular doses of alcohol17:36
mgagneAjaeger: glad to know I'm not the only one losing sleep over it =)17:36
ruhefungi: did you fix everything again? :)17:36
ruhefungi: here is
*** rgerganov has quit IRC17:36
Ajaegermgagne: feel free to discuss with annegentle on #openstack-doc or on our doc mailing list17:36
fungiruhe: i did not, but i'm thinking mordred is also doing things (or else it magically fixed itself, which would be simultaneously thrilling and frightening)17:37
jeblairokay, that revert fixes the last of the broken puppet problems17:37
ruhefungi: anyway, thank you very much for helping to dig through this issue!17:38
mgagneAjaeger: hehe, it makes it harder for people to come up with "templates" such as "Installs and configures the <program-name> service for OpenStack (<program-code-name>)." You end up with ungrammatical sentences :P17:38
*** markwash has joined #openstack-infra17:39
nibalizerjeblair: woot17:39
Ajaegerruhe, fungi: Interesting 0.16.1 is on but not on
*** e0ne has quit IRC17:40
fungiAjaeger: the latter is no longer used17:40
Ajaegerfungi: Ah, didn't know...17:40
*** UtahDave has joined #openstack-infra17:40
fungiAjaeger: the first one is our new full pypi mirror, the second one is our old partial mirror17:40
*** dstanek is now known as dstanek_lunch17:41
nibalizerclarkb: a pupetboard challenger appeared today:
*** pballand has quit IRC17:41
Ajaegerfungi: thanks for the explanation17:41
lifelessdhellmann: want me to look again at whatever it was you invoked my name w.r.t. ?17:41
openstackgerritA change was merged to openstack-infra/config: Revert "Fixing deprecation warnings"
anteayaI've been in the cinder meeting and such, kinda lost, anything I can do to help?17:43
*** chuckC has quit IRC17:43
*** UtahDave has left #openstack-infra17:43
anteayanibalizer: given the work we put into puppetboard is it worth spending any time evaluating puppetexplorer at this point?17:44
anteayawell mostly your work into puppetboard17:44
*** zzelle has joined #openstack-infra17:44
*** pballand has joined #openstack-infra17:46
*** mbacchi has joined #openstack-infra17:46
*** openstackgerrit has quit IRC17:49
*** openstackgerrit has joined #openstack-infra17:50
Ajaegerfungi, jeblair the pypi issue seems to be fixed - 0.16.1 is now used for gating!17:50
*** denis_makogon_ has joined #openstack-infra17:50
*** denis_makogon has quit IRC17:51
*** denis_makogon_ is now known as denis_makogon17:51
*** dmakogon_ has joined #openstack-infra17:51
clarkbnibalizer: neat. That seems more data query driven and less dashboardy17:51
* Ajaeger just cloned the new docs-specs - thanks jeblair!17:53
*** annegent_ has joined #openstack-infra17:54
*** otherwiseguy has joined #openstack-infra17:54
*** Shrews has quit IRC17:55
*** marcoemorais has joined #openstack-infra17:55
*** marcoemorais has quit IRC17:56
*** marcoemorais has joined #openstack-infra17:56
dstufftmordred: what's up17:56
dstufftjust got back from the dentist17:56
fungidstufft: we discovered quite a lot of things that bandersnatch apparently didn't mirror from pypi.python.org17:57
dstufftdid bandersnatch say the mirror was successful17:57
anteayaAjaeger: so now you need an openstack-doc-specs-core group made up, yeah?
*** Sukhdev has quit IRC17:58
anteayaAjaeger: with at least one person in it, the ptl I assume17:58
openstackgerritKhai Do proposed a change to stackforge/python-jenkins: make JSON depth response configurable
*** marcoemorais1 has quit IRC17:58
Ajaegeranteaya: yes, please17:59
fungidstufft: yaql was the first one reported (we didn't get anything at all mirrored for it) but then it was pointed out that a new release of opensatck-doc-tools wasn't being picked up either17:59
fungidstufft: and yeah, bandersnatch runs reported no errors at all17:59
Ajaegeranteaya: annegentle is the ptl17:59
*** arnaud has quit IRC17:59
*** pballand has quit IRC18:00
Ajaegerbut you can add me as well and I'll add her ;)18:00
Ajaegeranteaya: can you do this or who can?18:00
anteayafungi: the doc-specs project need an openstack-doc-specs-core gerrit group, with the openstack-doc-ptl in it, please18:00
anteayaAjaeger: I can't, I was just clarifying18:00
jeblairclarkb, fungi, mordred, SergeyLukjanov, jhesketh:
Ajaegeranteaya: thanks for the help ;)18:01
anteayaAjaeger: sure :D18:01
dstufftfungi: sounds like a bandersnatch bug, if if it's reporting that the mirror was successful and it wasn't18:02
*** mestery has quit IRC18:02
dstufftfungi: might be related to the switch to json instead of xmlrpc18:02
fungidstufft: yeah, definitely seemed like a bug, just wasn't sure if you happened to know a good way to isolate and debug18:02
fungior potential workarounds18:02
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Cleaned up autoresize_width directive.
clarkbjeblair: lgtm18:03
*** otherwiseguy has quit IRC18:03
dstufftfungi: not offhand, you can delete the state file to get bandersnatch to to a complete sync instead of an incremental sync tho18:03
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added search interface
jeblairfungi: as a result of fixing the puppet bugs, all the acl changes just went through:
*** annegent_ has quit IRC18:03
fungiAjaeger: anteaya: the acl doesn't seem to have been applied to that project in gerrit18:03
Alex_Gaynorjeblair: for the livegrep stuff, I'm proposing a new system, but don't actually have th ekeys to the kingdom, does that mean I need to coax someone from the infra team into "owning" it?18:03
anteayafungi: how interesting18:03
fungijeblair: oh, great!18:03
Alex_GaynorPS: would love some reviews on that :-)18:04
anteayaAlex_Gaynor: not really owning it, just is it in a place they can fix18:04
anteayaand do we have enough docs to fix any brokeness18:05
anteayaand yay18:05
Ajaegerfungi: puppet was just run for it - perhaps it's not finished yet?18:05
jeblairAlex_Gaynor: yeah, once the change lands, you'll want to poke someone to actually spin up the node, but other than that, we're quite happy for you to continue to feel an ownership interest in it via puppet modifications :)18:05
anteayaAlex_Gaynor: do you have the url for the patch?18:05
*** nati_ueno has joined #openstack-infra18:05
Alex_Gaynorjeblair: sounds good, thanks18:05
anteayayay livegrep!18:05
*** packet has quit IRC18:06
fungianteaya: Ajaeger: yeah, that's what i thought too, but manage-projects seems to no longer be in progress so it must have thought it did what it needed. perhaps another run will fix it, or produce some error message i can act on18:06
Ajaegerfungi: ok, let's wait and monitor ;)18:06
clarkbjeblair: fungi mordred I will kick off image rebuilds shortly to complete the pypi mirror switchover unless you want ot wait for further bandersnatch debugging18:07
*** shayneburgess has joined #openstack-infra18:07
*** changbl has joined #openstack-infra18:07
anteayaAlex_Gaynor: was there ever any discussion bringing livegrep under stackforge and therefore git.o.o?18:07
clarkbas those builds should complete now with the projects that were just created18:07
anteayaAlex_Gaynor: and do you have any thoughts on this?18:07
Alex_Gaynoranteaya: I don't think the author of livegrep really has any interest in moving it under stackforge, he's not involved in openstack18:08
*** sparkycollier has quit IRC18:08
anteayaAlex_Gaynor: okay18:08
anteayaAlex_Gaynor: just thought I would ask18:08
clarkbalso it is Go :P18:08
clarkbthough with trusty maybe that isn't so bad anymore18:09
clarkbGo on precise was funky18:09
fungijeblair: thanks! i'll see if i can find the corresponding gerrit service logs for that api call18:09
fungihopefully there's something useful18:09
clarkbaiui Go on precise reached a point where it worked for juju so they stopped updating it?18:09
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Moved template files into related modules
*** mestery has joined #openstack-infra18:10
*** ociuhandu has quit IRC18:11
anteayaclarkb: ha ha ha18:11
anteayathe works for juju standard18:12
fungihere's what the gerrit error log has...
*** Shrews has joined #openstack-infra18:12
*** sparkycollier has joined #openstack-infra18:13
*** pballand has joined #openstack-infra18:13
anteayafungi: a problem with threads?18:13
*** mestery_ has joined #openstack-infra18:14
*** mestery has quit IRC18:14
jeblairhrm it's there now18:15
fungiyeah, weird18:15
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Cleaned up autoresize_width directive.
jeblairfungi: i'm going to run m-p for that project, ok?18:16
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Renamed module folders for consistency
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Tag Complete directive has proper focus behavior
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Moved template files into related modules
fungijeblair: fine by me... i'm digging in a copy of its git repo now18:16
*** changbl has quit IRC18:17
anteayaAlex_Gaynor: question here:
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added focus directive
anteayaAlex_Gaynor: what was the criteria for the selected repos?18:17
jeblairfungi, mordred: same error again:
Alex_Gaynoranteaya: There wasn't one. It was the handful of projects I remembered off hand. We should totally add more stuff.18:18
*** marun_afk is now known as marun18:18
anteayaah okay18:18
anteayaAlex_Gaynor: is there any limit on how many repos we can grep?18:18
anteayawould be awesome if we could grep devstack and infra stuff too18:18
*** milina has joined #openstack-infra18:19
*** amitpp has joined #openstack-infra18:19
*** devkulkarni has joined #openstack-infra18:19
*** milina has quit IRC18:19
Alex_Gaynoranteaya: Not really, at some scale there's a RAM issue, but I don't think we're close to that18:19
anteayaAlex_Gaynor: this is our canonical list of openstack repos:
*** packet has joined #openstack-infra18:19
anteayaAlex_Gaynor: kk, I'll let you decide which ones you want to add18:20
*** ChanServ sets mode: +o jeblair18:20
mordredjeblair, clarkb: catch me up - was on the phone -- where are we with the mirror?18:20
Alex_Gaynoranteaya: Do you mind if I just wait until all the rest of the issues are dealt with before trying to update teh list?18:20
anteayabet it won't take long to find out where the RAM issue kicks in18:20
anteayaAlex_Gaynor: oh sure18:20
anteayaonce the module is in an up, should be easy to add more repos18:20
*** jeblair sets mode: +b *!*213.140.5@*18:20
anteayaAlex_Gaynor: I can make a note in the patch18:20
Ajaegeranteaya: I see that docs-specs is not in the canonical-list - who normally adds a repo there? Shall I send a patch18:21
*** jeblair sets mode: -o jeblair18:21
anteayaAjaeger: send a patch18:21
anteayathe tc has a chat about it and takes a vote18:21
anteayashould be an easy merge18:21
clarkbmordred: I think dstufft may have suggested doing another full mirror run18:21
*** ChanServ sets mode: +o jeblair18:21
*** mestery_ has quit IRC18:21
anteayait wasn't listed since we just created it18:21
*** sparkycollier has quit IRC18:21
dstufftit sounds like a legit bandernsnatch bug though18:21
*** mestery has joined #openstack-infra18:21
anteayaAjaeger: and yeah, additions to the programs.yaml file are all by hand18:21
Ajaegeranteaya: will do18:22
mordreddstufft: is this related to the stale mirror thing at all?18:22
fungimordred: so, for some strange reason, the missing directories/packages spontaneously appeared on the mirror after we talked about it, but i didn't run/do anything. i assumed you added them (manually or something)?18:22
dstufftmordred: could be, though it shouldn't be. bandernsatch should not have a successful mirror run until it's fetched all the packages18:23
*** jeblair sets mode: -b *!*213.140.5@*18:23
dstufftfungi: are you sure the previous runs were successful?18:23
mordreddstufft: I'm not sure it had a successful mirror run18:23
dstufftbandersnatch will retry to download things it couldn't download18:23
*** jeblair sets mode: +b *!*@213.140.518:23
*** jeblair sets mode: -o jeblair18:23
dstufftuntil it's donwloaded everything it thought it should18:23
dstufftthen it'll be a successful run18:23
mordreddstufft: this is what I'm poking at - when you say "unsuccessful" ... do you mean bandersnatch will stop processing thigns other than the thing that stale?18:23
dstufftno, bandersnatch will keep going18:24
dstufftit's greedy18:24
fungimordred: dstufft: it wasn't generating errors... can it be silently unsuccessful or something?18:24
*** shayneburgess has left #openstack-infra18:24
dstufftit'll fetch as much as it can, even if something is unavailable it'll keep going and get everything else18:24
*** pballand has quit IRC18:24
mordredthis is what it was giving us - but we still weren't getting yaql18:24
*** shayneburgess has joined #openstack-infra18:25
dstufftI might be slightly wrong then18:25
dstufftI know it's greedy for things like connection errors18:25
dstufftmaybe it's not for stale18:25
openstackgerritRyan Carey proposed a change to openstack-infra/jenkins-job-builder: support string values for bool fields "true", "false", "True" or "False"
dstufftmordred: is it still doing that?18:26
fungimordred: worth noting, the bandersnatch log says it suddenly sync'd yaql at 2014-07-02 17:11:28,874 (and since then we've been serving it)18:26
*** mestery has quit IRC18:26
mordredfungi: strange18:26
fungiit's as if it didn't previously realize that package existed on pypi at all18:27
dstufftwhat's your config file look like18:27
mordredfungi: I'm afk-ish right now - is the skil error still happnening?18:27
*** mestery has joined #openstack-infra18:28
fungimordred: apparently not. last mention of skll was 2014-07-02 17:05:02,62218:28
mordredfungi: ok18:29
mordredso - current working theory is that bandersnatch is not being greedy for stale things18:29
fungiso perhaps that was blocking completion?18:29
mordredbecause yaql was just uploaded today18:29
mordredso it makes sense that if bandersnatch decided to be non-greedy in the case of a stale thing, that we wouldn't get it18:29
mordreddstufft: what was teh curl command you said we could use to trigger resyncing stale things?18:30
fungiexplains why today's new openstack-doc-tools release wasn't getting mirrored either18:30
* mordred thinks we should put in a log filter looking for "Stale serial for package (.*)" and then do curl -X SYNC$1 if we see it18:31
mordredor whatever the curl command is18:31
*** chuckC has joined #openstack-infra18:31
mordredand by log filter, I mean we should change the cronjob to be "bandersnatch mirror |" - since that output should usually be very short18:32
mordredand by do that, I mean until we can land a patch upstream to do that directly18:32
mordredspeaking of - I need to fix my current pull requesst18:32
dstufftso I see what's happening18:32
clarkbhurray dstufft to the rescue18:32
dstufftbasically when bandersnatch runs, asusming the last run was successful (or it's your first run), it builds up a list of packages that need to be synced, and saves those to a todo list18:33
dstufftit then attempts to fetch all of those, greedily18:33
dstufftif any of them fail, it marks the run as unsuccessful, but it records which ones it didn't sync, and at what point in history it was trying to sync18:33
dstuffton the next run, if it sees that the last run was unsuccessful, it tries to complete that run18:34
dstufftit doesn't look for new things to sync, until it's already completed the in progress run successfully18:34
dstufftso in the case of a stale serial, you'll end up with everything but the stale serials completed, and it'll just fail each time until the cached page is no longer stale18:35
dstufftand which point it'll start a new runwhere it'll grab a new list of packages to sync18:35
dstufftmake sense?18:35
*** hashar has joined #openstack-infra18:35
*** mestery_ has joined #openstack-infra18:35
*** mestery has quit IRC18:35
clarkbthat seems like a bad case of premature optimization18:35
dstufftwhich part18:36
clarkbthe I am going to try the bits I know fail over and over until they don't fail18:36
dstufftwell you have to keep retrying them, because if you just skip them then the mirror will be missing packages forever18:36
dstufftmirroring doesn't work like rsync where it compares two trees and makes them equal18:37
dstufftit's a journal18:37
dstufftlike a database18:37
clarkbsure but you are skipping everything else in the mean time18:37
clarkbwhen you know you will probably fail18:37
dstufftcorrect, it could move the "target serial" out18:37
*** dims__ has quit IRC18:37
dstufftso that it just expands the range it's trying to sync18:37
dstufftbut that's more effort and other than a stale serial the most likely cause for failure is a network hiccup that goes away on the next run or two18:38
dstufftwhich is probably why it wasn't done18:38
dstufftbecause the issues (other than a stale serial) are largely self resolving18:38
mordredyah. so if we can trigger fixing stale packages, then that should also become more like network hiccups too18:38
jeblairdstufft: skll was in our logs for a _while_18:38
dstufftjeblair: I said other than stale :)18:38
mordreddstufft: what's the curl to fix that?18:39
jeblair(24 hours)18:39
jeblair(at least)18:39
dstufftFastly sits in front of PyPI and it caches all of our pages, bandersnatch knows what the version of that page _should_ be (it's in a header) and if the version it gets from fastly is older it fails with the stale message18:40
dstufftnormally PyPI tells fastly to purge the cache when something gets updated, but our worker doesn't retry failed tasks and is janky so sometimes a package gets missed :/18:40
dstufftit takes 24 hours for it to fall out of the cache naturally (sans grace periods if PyPI is down)18:40
*** devkulkarni has quit IRC18:41
dstufftand to answer mordred's question, anyone can purge a particular url just by doing curl -X PURGE
dstufftit doesn't require auth or anything18:41
*** amitpp has quit IRC18:41
dstufftthat'll instruct the Fastly varnish to drop the cached objects for that url18:41
dstufftin this case, the log lines looked like -> 2014-07-02 16:40:02,475 DEBUG: Expected PyPI serial 1143447 for request but got 1143446, so you'd do curl -X
clarkbshould bandersnatch try that instead of just failing?18:42
clarkbwould that be a good patch to write?18:43
dstufftI'd rather not have every single bandersnatch mirror issue a purge, the real solution is probably to stop using RQ and switch to celery so we can have the purge task retry as many times as needed until it actually successfully purges (and likely some sort of alert if it wasn't able to purge within some time frame)18:44
*** nati_ueno has quit IRC18:44
dstufftit's ok for y'all to do it for now as a work around though18:45
*** nati_ueno has joined #openstack-infra18:45
*** NikitaKo_ has joined #openstack-infra18:46
dstufftyou can tell when the last successful mirror run was on your mirror by looking at the last modified file btw18:46
clarkbso it was just now18:47
*** devkulkarni has joined #openstack-infra18:47
clarkbthat means everything should be happy?18:47
*** dims__ has joined #openstack-infra18:47
*** james_li has quit IRC18:49
clarkband we have 0.25 of skll now in both sdist and wheel form18:49
clarkbfungi: mordred ^ are you guys happy with that?18:49
clarkband should I work on image rebuilds now?18:50
clarkbI can do that while I do jenkins07 after lunch18:50
clarkbbecause somehow it is time for food nowish already18:50
fungidstufft: thanks for the explanation! that makes a lot more sense now18:52
*** mfer has quit IRC18:53
fungiso basically if someone complains that the mirror is behind, we can check the bandersnatch log, see what's breaking it, issue the purge call for it to fastly, then make sure the next run is successful18:53
*** nati_uen_ has joined #openstack-infra18:53
dstufftfungi: ya, I don't mind fi y'all do that automatically either tbh, having just openstack do it is better than ever bandersnatch mirror, and once I fix it so that there are retries and alerts I can bug y'all to stop doing it18:54
openstackgerritafazekas proposed a change to openstack-infra/elastic-recheck: Add query for pypi index fetching issues
*** chuckC has quit IRC18:54
dstufftFWIW it'll fix itself too18:54
*** NikitaKo_ has quit IRC18:54
dstufftit'll take a max of 24 hours to do so though18:54
dstufftI should probably lower that until retries are working18:55
fungidstufft: given how briefly we've been running it so far, i don't feel like we have much of a representative sample of this sort of breakage... what sort of frequency can we expect for it?18:55
*** dstanek_lunch is now known as dstanek_lunch_zz18:55
*** james_li has joined #openstack-infra18:55
dstufftfungi: dunno, I don't really pay attention to the mirrors unless someone reports a problem18:55
fungidoes this happen often? (on average once a month? a week?)18:55
dstufftif it happens often let me know and I'll prioritize moving to celery18:56
funginoted. thanks again!18:56
dstufftas far as I know it's not often, at least people don't complain about it very often18:56
*** nati_ueno has quit IRC18:56
dstufftif that's just becuase they deal with it idk18:56
*** chuckC has joined #openstack-infra18:57
fungigiven our ci workflow, we often release a new version of something to pypi and then expect to start (nearly) immediately consuming it from our mirror, so i suspect we'll start to notice fairly quickly if this is a frequent issue18:57
dstuffttheortically, with how PyPI's caching is set up, unless a purge failed, the speed at which you get a new package should only be limited by how often you run bandersnatch18:58
dstufftthe only thing on the worker queue is purges18:59
dstufftand a successful purge takes ~500ms or so18:59
*** changbl has joined #openstack-infra18:59
*** harlowja is now known as harlowja_away18:59
*** dstanek_lunch_zz is now known as dstanek18:59
*** jdahlin has joined #openstack-infra19:00
anteayapaste.o.o is 500ing for me19:01
pleia2anteaya: how long has it been since you've used it?19:01
anteayatwo minutes prior19:01
*** weshay has quit IRC19:01
fungianteaya: seems to be something to do with session cookies not getting stored in its db. i've noticed when that happens to me, if i clear all cookies for in my browser and refresh the url it works again19:01
pleia2ah ok19:01
anteayaand it is up now19:02
anteayaI can clear cookies19:02
pleia2yeah, I've had the same session cookie issue, but usually after I haven't used it for a while19:02
*** harlowja_away is now known as harlowja19:03
*** anode has quit IRC19:03
*** marcoemorais has quit IRC19:04
*** marcoemorais has joined #openstack-infra19:05
*** marcoemorais has quit IRC19:05
*** marcoemorais has joined #openstack-infra19:05
anteayaI just cleared cookies and refreshed a paste19:05
anteayathe paste if 500ing19:05
anteayaI'm up for trying something else19:06
anteayawhat else we got?19:06
anteayait doesn't usually 500 this long19:06
fungiwhat's the paste url you're failing on?19:06
anteayausually 3 or 4 tries gets it19:07
fungiyeah, works fine for me at the moment. i'll try clearing my cookies and see if it breaks for me now19:07
anteayaback now19:07
openstackgerritDoug Hellmann proposed a change to openstack/requirements: Add new library oslo.i18n
anteayanot sure what is happening19:07
fungiseemed to still be fine for me19:08
anteayamy system maybe?19:09
fungicould have been a brief issue with its trove backend or something too19:09
anteayawell moving on then19:10
*** chianingwang has joined #openstack-infra19:11
*** rcarrillocruz has joined #openstack-infra19:16
*** weshay has joined #openstack-infra19:17
*** rcarrill` has quit IRC19:18
nibalizerjeblair: so i'm feeling less than ready to go for 1pm. The review to add puppet3 master has some merge conflicts (especially in Are you anticipating a precise p3 master or a trusty p3 master?19:20
*** Sukhdev has joined #openstack-infra19:21
hasharnibalizer: have you tried our puppet catalog compiler ? :D19:24
nibalizerhashar: no, not yet19:24
nibalizerim just talking about the stuff to bring up a p3 master19:24
nibalizerwhich is mostly shell19:24
nibalizerbut i wrote it like months ago and it has a bit of bitrot going on19:25
*** devkulkarni has quit IRC19:25
hasharand no code comment :D19:25
*** otherwiseguy has joined #openstack-infra19:25
nibalizerhashar: also the catalog compiler differ thing is a bit too complex for me to be like 'oh i see how this works, boom, installed'19:25
hasharnibalizer: yeah you should reach out to Giuseppe _joe_ , he can guide you19:26
hasharit was really a one man / one shot effort19:26
hasharjust thought you might have interest in using it19:26
hasharfeel free to discard :-D19:26
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
*** mfer has joined #openstack-infra19:27
nibalizerya, have you or he blogged about it?19:27
nibalizeri think the puppet community shoudl be made a little bit more aware19:27
*** devkulkarni has joined #openstack-infra19:28
Ajaegerfungi: docs-specs looks fine to me, seems you got it running - now only the group is missing...19:29
fungijeblair: mordred: for whatever reason, the initial project creation run for openstack/docs-specs failed to create the groups file in the local repository. i created the group through the webui and manually created the commit for the acl and groups file (with the appropriate uuid) and gerrit let me push them just fine once i had the formatting correct19:29
*** pballand has joined #openstack-infra19:29
mordredfungi: everything about that makes me sad19:29
fungiAjaeger: who's supposed to be in openstack-doc-specs-core?19:29
*** mrmartin has joined #openstack-infra19:31
*** sigmavirus24 has joined #openstack-infra19:31
fungiAjaeger: i've added annegentle and made it visible now19:31
Ajaegerfungi: annegentle for sure - and looking at also Sean Roberts, Bryan Payne and myself.19:31
Ajaeger(at minimum)19:32
Ajaegerfungi: if annegentle can add more, it should be fine - thanks!19:32
fungiAjaeger: i added you and you and she can add whoever else you need19:32
Ajaegerthanks a lot, fungi!19:32
*** anode has joined #openstack-infra19:32
*** MarkAtwood has quit IRC19:33
*** mmaglana has quit IRC19:33
*** dims__ has quit IRC19:33
*** mmaglana has joined #openstack-infra19:34
dhellmannlifeless: you had questions on the spec for cross-project testing -
fungiclarkb: you might be interested by
*** mestery_ is now known as mestery19:38
openstackgerritKashyap Chamarthy proposed a change to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755
uvirtbotLaunchpad bug 1336755 in ceilometer " test times out waiting for Nova notification" [Undecided,New]
*** dims has joined #openstack-infra19:44
AjaegerAlex_Gaynor, dhellmann: could you review, please?19:46
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for Ceilometer timeout - bug 1336755
uvirtbotLaunchpad bug 1336755 in ceilometer " test times out waiting for Nova notification" [Undecided,New]
*** moted has quit IRC19:47
*** rcarrill` has joined #openstack-infra19:47
fungihowdy mrmartin19:48
mrmartinguys, how can I check reports for a specific day?19:48
*** dims_ has joined #openstack-infra19:48
dhellmannAjaeger: +219:48
fungimrmartin: i think puppetboard is probably not great for looking at historical reports...19:48
funginibalizer: clarkb: ^?19:48
mrmartinsomething happened inside the groups-dev, and I think it could be some race-condition in my puppet config19:49
SlickNikSukhdev / anteaya: I was following the steps in the google document but ran into some issues compiling the particular fork of gerrit-trigger-plugin that the doc mentions.19:49
Ajaegerthanks, dhellmann19:49
SlickNikSukhdev / anteaya:
*** rcarrill` has quit IRC19:49
*** praneshp has quit IRC19:49
nibalizerfungi: for getting the stuff from way-back?19:49
*** dims has quit IRC19:49
*** dkranz has quit IRC19:49
*** rcarrillocruz has quit IRC19:50
SlickNikSukhdev: Any ideas if you've seen this before, or what I may be running into?19:50
funginibalizer: yes, for browsing reports older than the 10 most recent19:50
*** Ryan_Lane has quit IRC19:50
nibalizeroh thats fixed19:50
nibalizerwe could upgrade our puppetboard and now there is a config option to say how many reports you want to see19:50
fungimrmartin: how far back were you wanting to look?19:50
mrmartinJul  1 23:4519:51
mrmartinthe reports around this date.19:51
fungiokay, so a little less than a day ago19:51
*** rcarrillocruz has joined #openstack-infra19:51
mrmartinI had a symlink there that was overwritten by a directory, and I guess somehow my symlink resource and apache manifest could collide in some special case.19:51
fungimrmartin: have you looked in the syslog on the server to see if puppet logged the issue there?19:51
*** marcoemorais has quit IRC19:52
mrmartinfungi: not yet, but I'm doing it now19:52
*** marcoemorais has joined #openstack-infra19:52
anteayaSlickNik: I just found Sukhdev in -neutron and explained the issue, I am assuming he is looking over the stacktrace19:52
*** moted has joined #openstack-infra19:53
anteayaSlickNik: he said he is heading out to lunch and will find you19:53
russellbso, "IBM PowerKVM Testing" has been completely broken for a good while now, can the account be disabled?  for example:
anteayaSlickNik: I'm not sure how long "lunch" is19:53
*** praneshp has joined #openstack-infra19:54
anteayaArxCruz: ^^19:54
SlickNikanteaya: Thanks! I'll be around.19:54
SukhdevSlickNik, anteaya: I was on my way to lunch  - let me look into this in bit and will let you know my findings….will you be around for a while?19:54
*** dims_ has quit IRC19:54
*** dprince has quit IRC19:54
SlickNikSukhdev: Yes, I will. And thanks much for helping with this!19:54
anteayaArxCruz: I thought we had talked about that this morning19:54
*** dims has joined #openstack-infra19:54
anteayaArxCruz: devs don't like being guinea pigs for testing19:54
SukhdevSlickNik: No worries!!!19:54
mrmartinfungi: it is the case, a collision between apache and my symlink:
jeblairnibalizer: trusty if we can swing it19:55
anteayaArxCruz: please disable your system until it is fixed19:55
ArxCruzrussellb: anteaya okay, I'm disabling it now19:55
anteayarussellb: thank you19:55
russellbArxCruz: thank you!19:55
anteayarussellb: sorry about that, they had emailed me yesterday to say they were having issues19:55
jeblairmrmartin: we attempted to upgrade the apache puppet module; failed, and downgraded it again.  not sure if that has a bearing on your issue19:55
anteayaI was off, and followed up wtih them this morning19:55
nibalizerjeblair: okay19:56
anteayarussellb: I didn't realize they were still posting broken results19:56
mrmartinjeblair: thanks, it explains the behaviour19:56
nibalizeri think i've gotten to a point where it will work well19:56
nibalizeri also want to pin to 3.4.* and we can go higher later if we want. objections?19:56
russellbanteaya: all good :)19:57
anteayarussellb: thanks19:57
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
*** rcarrill` has joined #openstack-infra19:58
mrmartinand how can I avoid similar situations in the future. The core problem, that apache manifest creates a vhost directory (call it w/), and later I need to replace this with a symlink that points to the actual release (w->slot1 or w->slot1). Anyone now a good puppet pattern for that?19:58
*** dims_ has joined #openstack-infra19:58
anteayaand I have more electrical storm19:59
anteayaI'm unplugging everything and hope I don't lose anything else this round19:59
*** rcarrillocruz has quit IRC20:00
nibalizerjeblair: so with that final state of the review for 83678 i think i'm good20:00
nibalizerit might even work on trusty :P20:00
*** dims has quit IRC20:00
*** marcoemorais has quit IRC20:01
*** marcoemorais has joined #openstack-infra20:02
*** marcoemorais has quit IRC20:02
jeblairnibalizer: okay, taking a look20:02
*** marcoemorais has joined #openstack-infra20:02
hasharclarkb: zaro: hey! I would like Marc Abramowitz to be added to python-jenkins. I have mailed -infra and -dev last week and you both seem to agree.  I am wondering what is the next step, does that need a puppet change?20:02
*** marcoemorais has quit IRC20:03
clarkbhashar: it is not a puppet change, instead we just need to add folks to gerrit groups20:03
*** marcoemorais has joined #openstack-infra20:03
hasharshould I be bold and just do it then reply back on the original email proposing the nomination?20:03
*** ociuhandu has joined #openstack-infra20:03
clarkbfungi: thanks, I will -1 it now with a link to the change telling people to not use that version20:03
clarkber not change to the mail thread20:03
clarkbhashar: that wouldn't bother me but jeblair may want to do it20:04
jeblairhashar: go for it :)20:04
hashardoing so thanks!20:04
jeblairclarkb: want to take a quick look at ?20:04
*** dims_ has quit IRC20:05
*** marcoemorais has quit IRC20:05
*** marcoemorais has joined #openstack-infra20:05
*** dims has joined #openstack-infra20:06
hasharjeblair: also JJB could use Darragh Bailey and Marc Abramowitz (again) as core reviewers.  +1 ed by a whole bunch of people including mighty Monty20:06
*** pballand has quit IRC20:06
jeblairhashar: thx, i'll finish that up this week20:07
fungiclarkb: yep, was more just thrilled that the mina-sshd update seems to have done the trick20:08
clarkbfungi: awesome. so ya I said in the review that it is a soft -1 and I think it would probably be ok to use 2.8.6 for git review testing20:08
fungiwill mark wip pending 2.8.720:09
clarkbfungi: my big concern there being 2.8.6 is essentially a dead version that no one should use20:09
clarkbwhich makes it less than ideal for testing20:09
fungiclarkb: 100%agree20:09
*** harlowja is now known as harlowja_away20:09
*** thedodd has joined #openstack-infra20:10
*** d34dh0r53 has joined #openstack-infra20:12
*** weshay has quit IRC20:12
*** weshay has joined #openstack-infra20:12
clarkbjeblair: nibalizer two small things noted inline20:12
*** pballand has joined #openstack-infra20:13
*** Sukhdev has quit IRC20:13
*** radez is now known as radez_g0n320:13
nibalizerkk will fix20:14
clarkbok I am going to start spinning up image rebuilds now20:15
clarkbthen I am going to remove envinject from jenkins07 to see what happens20:16
nibalizerclarkb: can you look back at I think you misread on the first comment?20:19
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
nibalizerthere ^^ is a fix for the second issue20:21
clarkbnibalizer: you are missing the period20:21
clarkbnibalizer: because 1.* != 1*20:21
*** UtahDave has joined #openstack-infra20:21
*** Ryan_Lane has joined #openstack-infra20:22
*** james_li has quit IRC20:22
nibalizerit doesn't have a * ?20:22
*** james_li has joined #openstack-infra20:22
nibalizerdoes it need to be '1.*' ?20:22
arosenI got a silly question here: I'm getting some pep8 errors but it seems like we never enforce these in any of the openstack projects. Just curious if there is something i need to change in the cookiecutter template?20:22
arosenLooking at .tox.ini it seems the same as python-neutronclient which doesn't enforce these.20:23
clarkbnibalizer: yes beacuse 1.* means match only version 1.x. But 1* matches versions 1,10,11,12,et al20:23
nibalizerokay i didn't see that the * is added in by the template20:23
nibalizerokay i'll fix it all up20:23
clarkbarosen: check the flake8 section in tox.ini?20:23
clarkbwe should declare war on editors that don't put newlines at the end of files though20:23
openstackgerritMarton Kiss proposed a change to openstack-infra/config: Add instance
nibalizerso doesn't that mean the 2.7 shoud be 2.7. on the line above?20:24
jeblairclarkb, arosen: cookiecutter bizarrely strips newlines from its templates20:24
clarkbnibalizer: possibly. I was more interested in keeping the pin file contents identical to the old contents20:24
clarkbnibalizer: and the old one uses 2.7* iirc20:24
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master
nibalizerso does 2.7 match 2.7.26?20:25
clarkbnibalizer: yes20:25
clarkbwell with the *20:25
nibalizeri guess it must, since it works everywhere20:26
nibalizeryea 2.7* is what I meant20:26
arosenjeblair: ah i see interesting. thaks!20:26
*** Hal_ has quit IRC20:26
clarkbjeblair: oh fun20:26
nibalizerokay so tthat changest adds 2.7.* as the pinning for default and 3.4.* as the pinning for hosts running 3. does that work or do you want it to be 2.7* on the old hosts?20:26
clarkbnibalizer: I am fine with 2.7.20:27
clarkbfungi: jeblair ^20:27
* clarkb reviews20:29
*** mestery_ has joined #openstack-infra20:30
*** alexpilotti has quit IRC20:31
mtreinishclarkb, fungi: you guys are still adding new projects on Fridays right?20:32
clarkbmtreinish: we add them whenever20:34
clarkbthe system works well enough that we don't do it on a particular day anymore20:34
jeblairexcept i won't be adding them this friday ;)20:34
mtreinishclarkb: ahh, ok20:35
mtreinishjeblair: heh, fair enough :)20:35
*** harlowja_away is now known as harlowja20:39
Ajaegerjeblair, clarkb: This patch was approved and needed a rebase: I'd love to see it go on soonish if you have some minutes later...20:40
* clarkb looks while fighting nodepool image builds20:41
Ajaegerclarkb: the nodepool images are much more important ;) Thanks!20:42
jeblairnibalizer: 83678 aprvd20:43
*** marun is now known as marun_afk20:43
*** denis_makogon has quit IRC20:44
nibalizerjeblair: woot20:45
openstackgerritAaron Rosen proposed a change to openstack-infra/config: Enable gating for python-congressclient
* Ajaeger waves good-bye20:45
*** Ajaeger has quit IRC20:45
openstackgerritA change was merged to openstack-infra/git-review: Ensure username is set for all tests
openstackgerritA change was merged to openstack-infra/config: Add node def for puppet3 master
nibalizerjeblair: so im not 100% on what our plan should be20:48
*** shashank has joined #openstack-infra20:49
nibalizerideally we bring up the puppetmaster_ver=3 vm, then --three, then slave it to the other puppet master, and hope it runs well20:49
nibalizerif not, then we have to get creative20:49
jeblairnibalizer: why slave it?20:50
*** Sukhdev has joined #openstack-infra20:50
SukhdevSlickNik: Are you here?20:50
Sukhdevshashank: are you here?20:50
*** zz_gondoi is now known as gondoi20:50
*** krtaylor has quit IRC20:50
openstackgerritKhai Do proposed a change to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target
Sukhdevanteaya: hi20:51
*** marun_afk is now known as marun20:52
*** shashank has quit IRC20:52
*** miqui has quit IRC20:53
*** shashank has joined #openstack-infra20:53
SukhdevSlickNik: Looks like you are gone20:53
zaroclarkb: would you be able to help move this along?
SukhdevI have a Dr. appt and I have to go20:54
SukhdevSlickNik: shashank is looking at your error and will provide you help.20:54
nibalizerjeblair: well don't we need to slave it so that it picks up an ssl cert, and all the stuff in openstack_project::server that makes it an openstack-infra machine?20:54
SukhdevSlickNik: please reach out to him - while i am gone20:54
nibalizerit's possible, maybe a bit tricky, to have it bootstrap itself20:55
SukhdevI will be back online in couple of hours or so20:55
*** dims has quit IRC20:55
shashankSlickNik: Can you please change PluginCommentAddedEvent event = new PluginCommentAddedEvent("CRVW", "1");20:56
nibalizeri also need to make at least one more small commit, we need to set 'ca = false\nca_server = ci-puppetmaster.o.o' in the puppet.conf of only this puppet3 master, that way we have a centralized CA20:56
shashankon line 63 in gerrit-trigger-plugin/blob/master/gerrithudsontrigger/src/test/java/com/sonyericsson/hudson/plugins/gerrit/trigger/mock/DuplicatesUtil.java20:56
SlickNikSukhdev: Thanks!20:56
shashankto +    public PluginCommentAddedEvent(String verdictCategory,20:57
shashank+                                   String commentAddedTriggerApprovalValue,20:57
shashank+                                   String commentPattern ) {20:57
*** Sukhdev_ has joined #openstack-infra20:57
shashankSlickNik: Oops, sorry20:57
jeblairnibalizer: i thought the plan was to operate it in parallel, moving hosts over to it opportunistically20:57
*** signed8b_ is now known as signed8bit_ZZZzz20:57
shashankChange that to PluginCommentAddedEvent event = new PluginCommentAddedEvent("CRVW", "1", "*" );20:57
jeblairnibalizer: i'm not sure what it needs from ci-puppetmaster, actually20:58
*** atiwari has quit IRC20:58
SlickNikshashank: Thanks for looking at this! I'm in back-to-back meetings over the next two hours, I'll make the changes you suggest after that and let you know how it goes.20:58
shashankSlickNik: No problem20:58
*** atiwari has joined #openstack-infra20:59
nibalizerjeblair: you're correct that it will be acting in parallel. but we need to change the default puppet master behaviour20:59
nibalizerby default it will generate and manage its own CA for client certs21:00
*** Sukhdev has quit IRC21:00
*** dims has joined #openstack-infra21:00
nibalizerwhat we need to do is to tell it to run the puppet master service as normal, but to use ci-puppetmaster as the CA server21:00
*** yfried has quit IRC21:00
*** melwitt has quit IRC21:00
*** melwitt has joined #openstack-infra21:01
nibalizerwe'll sign the cert on puppetmaster.o.o with the ca on ci-puppetmaster.o.o and that will allow a rooter to hop on an arbitrary box and issue a 'puppet agent --noop --server=puppetmaster.o.o' and see if anything breaks21:01
*** melwitt has quit IRC21:01
*** melwitt1 has joined #openstack-infra21:01
nibalizerdoes that make sense?21:01
*** mrda_away is now known as mrda21:02
*** hogepodge has quit IRC21:02
*** Sukhdev_ has quit IRC21:02
annegentlemordred: aroo?21:03
*** yfried has joined #openstack-infra21:03
*** melwitt1 has quit IRC21:03
*** andreykurilin_ has joined #openstack-infra21:03
*** shashank is now known as shashankhegde21:04
*** hogepodge has joined #openstack-infra21:04
*** melwitt has joined #openstack-infra21:04
jeblairnibalizer: so the new puppetmaster will need a copy of ci-puppetmaster's cert in order to validate signed certs from clients?21:04
*** mestery_ has quit IRC21:04
*** ArxCruz has quit IRC21:04
*** rfolco has quit IRC21:05
jeblairnibalizer: or does "ca_server =" cause it to do that validation over the network?  (and if so, how?)21:05
*** adalbas has quit IRC21:06
*** atiwari has quit IRC21:06
clarkbok I am removing envinject plugin from jenkins07 now21:06
*** markwash has quit IRC21:06
*** gokrokve has quit IRC21:06
nibalizerjeblair: so the new puppetmaster will need its own cert, we can't re-use the cert from ci-puppetmaster21:07
*** mfer has quit IRC21:07
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: New Story Autofocus
jeblairnibalizer: yeah, that's not what i was trying to ask; i'm curious about how the new puppetmaster will validate the signatures of client certs presented to it21:07
nibalizerca = false tells the puppet daemon not to respond to certificate signing requests, if it gets one it just errors and closes the connection, thats how we prevent puppetmaster from running its own ca21:08
nibalizerca_server = ci-pupputmaster will tell puppetmaster that it needs to check in to that host to get a certificate21:08
jeblairnibalizer: oh, we should do that on ci-puppetmaster too -- we never accept signing requests21:08
*** gokrokve has joined #openstack-infra21:08
nibalizerno it does more than that, it turns off all certificate authority actions too21:08
nibalizerhow do you put certs out there21:08
nibalizerdo you puppet cert generate then copy?21:08
jeblairnibalizer: yes, because we launch them from ci-puppetmaster, so we have a trusted path for the copy21:09
nibalizerpuppetmaster.o.o will validate clients by checking to see if their cert has been signed by the ca.pem that puppetmaster.o.o got from puppetmaster-ci.o.o21:09
nibalizerclients perform a reciprocal validation21:09
jeblairnibalizer: okay, so checking the ca.pem from ci-puppetmaster is a side effect of setting ca_server=, right?21:10
nibalizerwell, no21:10
nibalizerif we're doing cert generate, copy, then there is no need for ca_server to be set21:10
nibalizerits only important if using puppet agent, then puppet cert sign to do the initial key setup21:11
nibalizerkey/cert setup, i should say21:11
nibalizeractually puppet doesn't /do/ any certificate validation, it hands all that off to apache21:11
jeblairnibalizer: ok, so what will cause puppetmaster to validate client certs are signed by ci-puppetmaster's ca.pem?21:11
jeblairor apache even :)21:11
nibalizerso its our job to put ci-puppetmaster.o.o's ca.pem file in the right place then point at it in apache and then we're golden21:12
nibalizerjeblair: apache configuration dirctives set that up21:12
jeblair./sites-available/puppetmaster:        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem21:12
openstackgerritKhai Do proposed a change to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target
jeblairthat one i guess :)21:12
nibalizerjeblair: so if you look in that file there is a section that allows unauthenticated clients to submit certificate signing requests21:13
*** dims_ has joined #openstack-infra21:13
nibalizerwe could presumably remove that to get the behaviour you spoke of a bit ago: rejecting certificate requests21:13
clarkbshutting down jenkins07 now21:14
jeblairnibalizer: i don't see that option immediately, but that might be nice, yeah.  but we can worry about it later21:14
*** mmaglana has quit IRC21:14
jeblairnibalizer: so should we just copy the ci-puppetmaster ca files (cert and key) over to the new puppetmaster?21:14
*** dims has quit IRC21:15
clarkband starting now21:15
*** julim has quit IRC21:15
nibalizerjeblair: no21:16
nibalizerwe should puppet cert generate puppetmaster.o.o , then copy those files over, as well as the ca.pem from the CA and the CRL21:16
nibalizeralso... crud, so trusty has apache 2.421:16
*** r1chardj0n3s_afk is now known as r1chardj0n3s21:17
nibalizerso we'll have to add logic to detect and handle that in our puppetmaster vhost template21:17
jeblairnibalizer: sorry, i was only asking about the ca.  so yes, agreed that the machine cert itself should be unique for the new server, but as for the CA -- copy it over completely?21:17
clarkb jobs are running and we still list parameters21:17
clarkbso hopefully this really does fix it :)21:17
*** gyee has quit IRC21:18
*** annegent_ has joined #openstack-infra21:18
nibalizerjeblair: no no, we can't copy it over21:18
nibalizerit needs to be unique21:18
*** dims_ has quit IRC21:18
nibalizerbecause every time you sign a cert with a ca it gets a serial number and those numbers are what is used to revoke it in the crl21:19
nibalizerso best pratice is to only have one if you can get away with it21:19
nibalizersecond best practice if you really need two machines, is to keep it on nfs or something so the files can never disagree21:19
*** james_li has quit IRC21:20
nibalizerbut we don't need that CA directory on puppetmaster.o.o it can live on ci-puppetmaster.o.o for the forseeable future21:20
jeblairnibalizer: okay, so just copy the ca cert for now, but not the key?  i mean, we'll have a ca on that host, it just won't be usuable in that state.21:20
*** mmaglana has joined #openstack-infra21:21
nibalizerca = false on puppetmaster.o.o tells pm.o.o to be a 'dumb' puppet master, just compiling catalogs and such, leaving cert signing to a different puppetmaster21:21
nibalizerjeblair: it won't have a ca at all, the only file you need to copy over that has ca in it is /var/lib/puppet/ssl/certs/ca.pem21:21
nibalizerbut thats just the root ca certificate, same file as is in /etc/pki and /etc/ssl and in your firefox21:22
nibalizerits only purpose is to validate that a certificate presented to you has been signde by that certificate authority21:22
nibalizerand its only purpose for pm.o.o is to make sure clients have really been signed by the CA on ci-puppetmaster.o.o21:22
nibalizeralso, is the puppetmaster apache vhost managed in puppet? I don't see it in infra/config anywhere21:23
jeblairnibalizer: i'm assuming something will create those files and set up a valid ca.  so at that point, i think it's likely we will have a broken ca because the key doesn't match the cert.  but we don't care because we don't plan to use it.21:23
openstackgerritBen Nemec proposed a change to openstack-infra/config: Use tripleo-core for dib-utils
jeblairnibalizer: i expect that it comes from the .deb21:23
*** signed8bit_ZZZzz is now known as signed8b_21:23
jeblairclarkb: erm, that screen was supposed to go away21:23
nibalizeroh the puppetmaster-passenger package, that makes sense21:23
clarkbjeblair: yes I think jenkins provides that screen21:24
nibalizerjeblair: yes, probably the post-install(and start) script of the puppemaster-passenger package will generate a ca21:24
clarkbjeblair: I don't think the plugin does21:24
jeblairclarkb: jenkins-dev does not have it21:24
nibalizerbut a combination of putting ca=false in the [master] section of puppet.conf and rm -fr /var/lib/puppet/ssl will take care of that21:25
clarkbjeblair: oh21:25
nibalizerthen you can copy over the four files you need from ci-puppetmaster.o.o21:25
clarkbjeblair: the plugin is definitely uninstalled21:25
*** abhishek__ has joined #openstack-infra21:25
abhishek__hi all21:26
abhishek__can any one review this patch?21:26
abhishek__thank you21:26
*** mmaglana has quit IRC21:26
jeblairclarkb: i wonder if 'rebuild plugin' is responsible?21:26
*** jp_at_hp has quit IRC21:27
clarkbjeblair: /me looks21:27
*** sgordon has quit IRC21:27
jeblairoh, hrm.  some jobs do have it...21:28
jeblairon jenkins-dev21:28
fungii've not been reviewing the puppetmaster^3 changes closely enough... how do we address hiera sync from the old master? entry at a time? wholesale copying of the hiera file?21:28
clarkboh weird21:28
clarkbjeblair: I think removal of that screen is less pressing21:29
clarkbit would have been nice to have but the more important thing is getting the env vars in without modifying them21:29
pleia2fungi: I had assumed wholesale copying21:29
jeblairclarkb: yeah21:29
pleia2shouldn't change that much21:30
jeblairpleia2, fungi: yeah, i think we probably just double our work there for a while :)21:30
pleia2some of our dependencies are a bit tangled up (referenceing gerrit pubkey on a few servers, for instance) so I think chopping it up for the move is more work21:30
jeblairnibalizer: ok so i think we know what we're doing now; do you want to write that second patch before i launch the new host?21:30
jeblairnibalizer: or go aheand and launch it, and fix it up with the update later?21:31
*** dims has joined #openstack-infra21:31
*** SumitNaiksatam has quit IRC21:32
clarkbarg I think centos6 builds are broken again due to facter21:32
clarkbfix incoming21:35
*** mrmartin has quit IRC21:36
openstackgerritClark Boylan proposed a change to openstack-infra/config: Readd facter-2* exclude on centos6.
nibalizerjeblair: as you pointed out21:36
clarkbjeblair: nibalizer ^21:36
nibalizerwhatever i do with ca = false is moot because the pacakge is gonna go do things anyways cus its 'helping'21:37
nibalizerim about to submit the patch21:37
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf
*** rcarrillocruz has joined #openstack-infra21:38
openstackgerritDavid Lyle proposed a change to openstack-infra/reviewstats: Updating Horizon core with recent changes
nibalizeralso i had some good early results from running beaker on localhost, nothing to share yet really, but hopefully by the end of the week something you all can see/poke at21:38
reedfungi, jeblair: when you have time, this is good news finally
jeblairreed: cool21:39
jeblairnibalizer: lgtm21:39
*** rcarrill` has quit IRC21:40
jeblairclarkb: lgtm21:40
jeblairclarkb: and i'll trade you :)21:40
*** signed8b_ is now known as signed8bit_ZZZzz21:41
openstackgerritA change was merged to stackforge/python-jenkins: Use pbr, pep8 checks & fixes, add "doc" tox target
*** ociuhandu has quit IRC21:42
clarkbjeblair: nibalizer that change failed testing21:44
*** signed8bit_ZZZzz is now known as signed8b_21:45
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf
openstackgerritA change was merged to openstack-infra/config: Readd facter-2* exclude on centos6.
nibalizerclarkb: ya, just posted a fix21:46
nibalizersorry i needed to verify that undef works how I think it does21:46
*** sweston has joined #openstack-infra21:52
*** chianingwang has quit IRC21:53
*** jamielennox|away is now known as jamielennox21:55
*** james_li has joined #openstack-infra21:55
*** abhishek__ has quit IRC21:57
zaromsabramo: noticed you approved your change own change.  Sorry I didn't make you aware, but I think we kinda have a policy against that.21:57
*** melwitt has quit IRC21:57
*** dangers is now known as dangers_away21:58
*** MarkAtwood has joined #openstack-infra21:59
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Allow site.pp to manage ca and ca_sever in puppet.conf
*** mbacchi has quit IRC22:04
openstackgerritSpencer Krum proposed a change to openstack-infra/config: Adding default value for hiera lookup
msabramozaro: Oops. sorry about that22:05
*** mmaglana has joined #openstack-infra22:06
msabramooh I see you pushed up patch sets but probably just to rebase it; it was originally me22:07
msabramosorry bout that22:07
*** rcarrillocruz has quit IRC22:08
*** eharney has quit IRC22:08
*** rcarrillocruz has joined #openstack-infra22:08
mattoliverauGuten morgen22:08
*** penguinRaider has joined #openstack-infra22:09
*** doug-fish has left #openstack-infra22:10
hasharmsabramo: hi! I have added you as a core reviewer to python-jenkins \O/22:11
msabramohashar: Yes, I saw and I am honored by that :-)22:12
hasharzaro: my bad, should have talked about the double CR+2 with Mark before adding him :/22:12
msabramothough apparently I immediately abused my power :-)22:12
msabramowhat happens now? military tribunal? :-)22:12
fungiguten abend mattoliverau22:13
hasharthe idea is to seek review from two other core contributors, one of them (or even a third one) would then formally approve22:13
*** andreykurilin_ has quit IRC22:13
hasharso you have some kind of a double check.22:13
hasharzaro: I can't afford anytime this week to review stuff onJJB.  Will definitely allocate half a day next week though, then I am on vacations22:14
mattoliveraufungi: I see you've been practicing too :P22:14
*** nelsnelson has quit IRC22:15
*** r1chardj0n3s is now known as r1chardj0n3s_afk22:15
*** d34dh0r53 is now known as mostly_d34dh0r5322:16
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module
hasharzaro: maybe python-jenkins could use a minor release just before that pbr patches that landed in22:19
hasharzaro: then bump version to 0.3 ?22:19
jeblairprobably a good idea22:19
zaromsabramo: tribunal would be interesting, but i think newbie pass is ok this time :)22:20
zarohashar: ++, will do tomorrow maybe.22:21
hasharzaro: left a note on the pbr change . Thanks!22:21
nibalizerjeblair: clarkb want to review ?22:21
hasharcrashing to bed. Have fun folks22:21
*** otherwiseguy has quit IRC22:21
nibalizerturns out you HAVE to use the <% -%> syntax if you dont' provide any string output22:22
zarohashar: although it probably should be major release since last release was a year ago and there seems to be lots of changes?22:22
jeblairnibalizer: is ready?22:22
nibalizerjeblair: yes22:23
*** gondoi is now known as zz_gondoi22:24
zarojeblair: have an opinion? should release be 0.3 or  0.2.2?22:24
*** melwitt has joined #openstack-infra22:24
openstackgerritDoug Hellmann proposed a change to openstack-infra/release-tools: Use python interpreter in the $PATH
openstackgerritDoug Hellmann proposed a change to openstack-infra/release-tools: Add requests requirement needed by spec2bp
jeblairzaro: i'd do 0.3.022:24
*** melwitt1 has joined #openstack-infra22:24
*** maxbit has quit IRC22:24
zarohashar, msabramo : ^ i agree with jeblair22:25
hashar0.3.0 without pbr so ?22:25
zarohashar: yes. then bump next one to 0.422:25
hasharsounds good to me22:26
zarohashar: goodnight22:27
hasharthank you!22:28
*** melwitt has quit IRC22:28
*** shashankhegde has quit IRC22:30
msabramoyeah I agree with 0.3.0 for next version of python-jenkins without pbr and then 0.4.0 for pbr22:31
clarkberr: /Stage[main]/Openstack_project::Thick_slave/Package[python-requests]/ensure: change from 1.1.0-4.el6 to absent failed: Execution of '/bin/rpm -e python-requests-1.1.0-4.el6.noarch' returned 1: error: Failed dependencies:22:32
msabramoeven without pbr, it changed quite a lot and changed maintainership and everything22:32
clarkbpython-requests is needed by (installed) cloud-init-0.7.4-2.el6.noarch22:32
clarkbI think yum is trying to tell me it couldn't upgrade python-requests because the version of cloud init that is installed dpeends on the version it already has?22:32
ianwjeblair: acquired a few more reviews; any more issues?22:32
anteayaso more rain, right now we are at 6.5 inches in 48 hours, not sure but it might break a record22:32
jeblairianw: aprvd22:33
clarkbanteaya: are you commuting by boat yet?22:33
clarkbjeblair: so should we move forward on envinject or do a bit more sluething to remove the parameters page first?22:33
clarkbjeblair: I guess we should also run a job on 07 to check that it really does fix the parameters22:33
ianwjeblair: thanks!22:34
jeblairclarkb: i'd move forward on that to solve the error and regroup on the parameters page since it's less critical22:34
clarkbok I am going to put 06 and 05 in shutdown mode now22:34
anteayaclarkb: no commute22:34
*** annegent_ has quit IRC22:34
anteayabut I am sizing up the available timber for the ark22:35
anteayaI should be in good shape22:35
anteayathe deck posts will be my beams22:35
clarkband done put in shutdown mode22:35
*** hashar has quit IRC22:36
*** otherwiseguy has joined #openstack-infra22:36
*** jgrimm has quit IRC22:37
*** zzelle has quit IRC22:37
*** melwitt1 has quit IRC22:38
jeblairclarkb: quick question on your zuul poll patch...22:39
jeblairclarkb: would it be possible to get the test cases from the other class?  it looks like just code duplication there...22:39
*** r1chardj0n3s_afk is now known as r1chardj0n3s22:40
*** signed8b_ is now known as signed8bit_ZZZzz22:41
openstackgerritA change was merged to openstack-infra/nodepool: Track last allocations to ensure forward-progress
clarkbjeblair: maybe?22:42
clarkbit has been a while I would haveto go reread22:42
clarkbI think the reason for that was we didn't want to run all of the tests from the other class22:42
clarkbjhesketh: jeblair
*** anode has quit IRC22:43
clarkbthat still isn't quite working right and it appears that maybe jenkins without envinject truncates the variable?22:43
clarkbso much :(22:43
*** anode has joined #openstack-infra22:44
*** pballand has quit IRC22:44
*** shashankhegde has joined #openstack-infra22:45
clarkbjeblair: I remember now. The sleeps are what is important22:45
jeblairclarkb: oh, ok.  thx.22:45
clarkbjeblair: those tests are different and have artificial waits that force the polling to occur22:45
clarkbsimilar but differnet22:45
jeblairclarkb: er, any chance we can make those a bit faster? :)22:46
clarkbwe have 1 second precision, so 2 seconds maybe?22:46
jeblairclarkb: for the timer tests i do tenths of a second22:46
openstackgerritA change was merged to openstack-infra/config: Move oslo default test node to trusty.
clarkbjeblair: I think that may end up being shorter than gits fetch time22:47
openstackgerritA change was merged to openstack-infra/config: Move trove default test node to trusty.
jeblairclarkb: if we depend on something happening in a sleep, then it's a racy test anyway22:47
clarkbjeblair: it shouldn't be racy because we open the flood gates after the wait22:48
clarkbthe self.update_repo() calls make it sanish22:48
openstackgerritMichael Krotscheck proposed a change to openstack-infra/infra-specs: StoryBoard Permissions Specifications
swestonclarkb:  fungi:  hello.  Would either of you have a few minutes to provide some direction on how to get console logs out of Jenkins?22:50
*** maxbit has joined #openstack-infra22:50
clarkbsweston: can you be a little bit more specific? what are you trying to do?22:50
swestonclarkb: sure.22:50
anteayahe is trying to get utc timestamps into logs for third party ci22:51
swestonclarkb:  trying to get the output for the dsvm-tempest-full job22:51
clarkbwe use a plugin to do that22:51
swestonis it the html publisher?22:51
clarkbone sec22:52
fungianteaya: you have my sympathies. next island south of us is currently under a mandatory evacuation order and this is only the first cyclone of the season (hopefully we get a pass on this one)22:52
clarkbsweston: is what we use at version 1.3.122:52
*** david-lyle has quit IRC22:53
jeblairclarkb: point being, even with sleep(3) we can't be certain it's actually polled and failed; what would be nice is looping with a small sleep until at least one poll failure has happened.  that probably means adding some instrumentation; possibly monkeypatching.  but then we can be assured the test dtrt, and can speed it up.22:53
*** devkulkarni has quit IRC22:54
anteayafungi: :(22:54
*** david-lyle has joined #openstack-infra22:54
anteayafungi: where do they go? and how secure are their houses when they evacuate?22:54
clarkbjeblair: sure. I think jhesketh asked for something ismilar but I don't think we do any of that in zuul's test frame work today22:54
anteayafungi: where would you go if you had to? you just arrived22:54
clarkbjeblair: its very functional testing and not mock testing22:54
jeblairclarkb: yeah, i don't want to change that22:55
*** dimtruck is now known as zz_dimtruck22:55
jeblairclarkb: however, we go to great lengths to functional test as quickly as possible22:55
fungianteaya: we "go west"22:55
*** krtaylor has joined #openstack-infra22:55
jeblairfungi: go far enough and you hit a desert :)22:56
clarkbjeblair: do you have suggestions on how you would do mock like functional testing here? monkeypatch in the poll functions with one that sets a flag?22:56
fungijeblair: indeed22:56
anteayafungi: got that wagon packed, just in case?22:56
*** msabramo has quit IRC22:56
anteayakrtaylor: hey there22:56
anteayakrtaylor: so can you come to germany yet?22:56
fungianteaya: we just go visit family in the mountains for a few days if it comes to it. not really a huge hassle22:56
anteayafungi: kk, glad you have a plan if need be22:57
*** camunoz_gone is now known as camunoz22:57
swestonclarkb: ok.  right now I am using version 1.5.12, and it is configured from the scp publisher defined in macros.yaml file here:
anteayaI'm on top of a hill so pretty safe22:57
anteayaI'm in the place everyone else comes to in times of bad things happening22:57
clarkbsweston: I think that newer versions may not work because they split the data out or something silly22:58
clarkbsweston: also you haev to set the format in your global config22:58
*** virmitio has quit IRC22:58
swestonclarkb: ah, ok.  I will look into that.22:58
anteayasweston: and keep notes please so you can share at a future meeting22:58
anteayaand so we can add to docs22:59
*** devkulkarni has joined #openstack-infra22:59
*** david-lyle has quit IRC22:59
clarkbok all of the rax images that matter have been updated22:59
clarkbhpcloud is in progress22:59
swestonclarkb: anteaya: yes, absolutely.  at some point I'd like to have a discussion with infra about how to adapt the infra's puppet manifests in the config repo to run third party infrastructure.  this is what I've been working on for the last several weeks.23:00
jeblairsweston: are you attending the qa/infra meetup in germany?23:01
*** ociuhandu has joined #openstack-infra23:01
*** blamar_ has joined #openstack-infra23:01
*** blamar has quit IRC23:01
*** blamar_ is now known as blamar23:01
swestonjeblair: no, unfortunately not.  may I somehow contribute to the discussion/meetup online?23:02
jeblairsweston: well, we'll probably have some specs come out of it; is a related first step23:02
jeblairsweston: but several folks are interested in that area, your help will be appreciated :)23:03
swestonjeblair:  sure, I would be happy to share what I have so far23:03
anteayamorning jhesketh23:04
swestonjeblair: I have all of my changes in github, when you are ready I will be happy to send the link23:04
jeblairsweston: great, best way to share is to submit patches to gerrit, and write/review specs23:04
*** amcrn has quit IRC23:04
swestonjeblair:  ok.23:04
*** blamar has quit IRC23:04
swestonjeblair:  suggestions on where/how to get started?23:05
clarkbjhesketh: hello23:05
clarkbjhesketh: please see
clarkbthat ran without envinject23:05
*** amcrn has joined #openstack-infra23:05
*** otherwiseguy has quit IRC23:05
anteayasweston: have you reviewed all open patches with topic:third-party ?,n,z23:06
swestonanteaya: no, first time I've seen these23:06
jheskethclarkb: yeah just saw that23:06
jheskeththat's kind of disappointing23:06
anteayasweston: great, let's start there23:07
jheskethclarkb: it looks like it's truncated at the \n23:07
swestonanteaya:  happy to.  I will start on the patch review this evening.23:07
*** sigmavirus24 has quit IRC23:08
jheskethclarkb, jeblair: Maybe for a work around for being unable to send multiple line variables to Jenkins I could patch zuul to also send "SWIFT_logs_HMAC_BODY_PART{1,4}"23:08
jheskethwhat do you think?23:08
anteayasweston: thanks23:08
anteayaI just checked the annual rainfall stats, we doubled the usual monthy rainfall for june in two days23:09
swestonanteaya: yw.  clarkb:, jeblair: thanks for the guidance.  looking forward to working with you all some more :-)23:09
anteayausual is 86 mm, or 3.25 inches23:09
jeblairjhesketh, clarkb: the "ZUUL_CHANGES" parameter that zuul sends separates them with '^' for that reason; back when gerrit-git-prep used it (a loooong time ago), it did some sed magic or something to split it on that char23:10
*** msabramo has joined #openstack-infra23:10
anteayafor a month23:10
jheskethjeblair: right, except the hmac body is in the format that the signature used23:10
jheskethand the format that swift expects23:11
jheskethso I'd like to keep sending the hmac_body as it is but then also provide a workaround for Jenkins23:11
jeblairjhesketh: yup, just adding information :)23:11
jheskethcool :-)23:11
*** annegent_ has joined #openstack-infra23:12
clarkbI would be on board with either approach23:12
clarkbI have 05 and 06 in shutdown mode still23:12
clarkbI don't see any reason to leave envinject in place as we wanted it gone23:13
clarkbso I will probably continue on my path of removing it23:13
jeblairclarkb, jhesketh: is upgrading envinject to the new version a solution?23:13
*** maxbit has quit IRC23:13
clarkbjeblair: yes I think that was an alternative23:14
*** whoops has quit IRC23:14
jhesketh:-( that was what I wanted to do last week23:14
jheskethwe've done well to remove the envinject stuff, I think adding it back in would be a step backwards23:14
jeblairjhesketh: well, that was based on the understanding that the problem was that envinject was corrupting data; that seems to be falso23:15
clarkbjeblair: it was corrupting the data23:15
jeblairwe definitely don't want to use it in jobs23:16
jheskethI'm actually unsure why we don't want it to be used?23:16
jeblairhowever, if it helps the transition to not-jenkins, i'm fine keeping it around23:16
jheskethor is it simply that we want to reduce our module consumption to tie us to jenkins less23:16
jeblairjhesketh: because it's a jenkins-specific way of describing a job23:16
jeblairjhesketh: yes23:16
jheskethI'm more inclined to have the workaround in zuul, but that would add 4 more variables to the gearman work data that may have performance effects23:17
phschwartz:( my brain hurts. This is the worst day I have had coding in about 12 years. I made the stupid decision to move monday from Vi due to non-stop plugin issue and sick of my 3k line config to emacs after not using it for almost 10 years. Then I had yesterday off for jury duty. Talking about making your life hell.23:19
clarkbphschwartz: the answer is to just stick with vim23:20
clarkbI had similar problems with vim on saucy crashing on yaml files due to segfault in regex engine23:20
SlickNikphschwartz: evil mode :)23:20
clarkbI tried evil but it is no where close enough23:20
jeblairjhesketh, clarkb: i think the issue may be in gearman-plugin23:20
clarkbtoo many things break23:20
clarkbjeblair: oooh23:20
jeblairi'll have more in a sec23:21
phschwartz clarkb: I have been having a mix of that and YCM dying non-stop on my 14.04 installs. It has been driving me insane.23:21
phschwartzclarkb: Which spawned the change.23:21
clarkbtrusty has been fine with me but I avoid plugins for hte most part23:21
clarkbtoo many habits from when I had to have working vim on all the platforms23:22
*** Sukhdev has joined #openstack-infra23:22
clarkbjeblair: interesting23:22
clarkbbut also a simplish fix23:22
clarkbI should probably continue to remove envinject then?23:23
clarkb05 and 06 are close to being done and I can do whatever23:23
*** alexandra has joined #openstack-infra23:24
*** rlandy has quit IRC23:25
*** alexandra is now known as asettle23:25
SukhdevSlickNik: ping23:25
jeblairclarkb, jhesketh: it's not yet clear to me if that will change behavior in our case23:25
clarkblooking at the logstash worker queue graph for today I want to declare success on that front and I should probably move on to do ssds on all the nodes23:25
clarkbjeblair: roger23:25
*** r1chardj0n3s has left #openstack-infra23:25
SlickNik Sukhdev: Spoke to shashank, and he offerred a workaround.23:25
SlickNikSukhdev: I haven't had a chance to try it out yet.23:26
*** annegent_ has quit IRC23:26
openstackgerritA change was merged to openstack-infra/config: Move troveclient default test node to trusty.
SlickNikSukhdev: Will do so this evening, and let you know how it goes.23:26
*** sabeen2 has quit IRC23:26
SlickNikSukhdev: Thanks so much for your help with this!23:26
SukhdevSlickNik: good - I discussed it with him before I left - he is very familiar with the code23:26
*** mugsie has quit IRC23:26
anteayaSlickNik Sukhdev shashankhegde thanks to all of you for working on this23:27
anteayaI really appreciate it23:27
SukhdevSlickNik: no worries -- let me know it goes and then we will take it from there23:27
*** james_li has quit IRC23:27
anteayaplease keep notes to so we can use them for getting this patch upstreamed please23:27
anteayaI'd like that to happen if possible23:27
jheskethjeblair: hmm, it seems like a good lead23:27
jeblairjhesketh, clarkb: i should be able to test it in a few mins23:28
Sukhdevanteaya: once SlickNik confirms the fix, I'll update the document so that others can benefit from it23:28
jheskethjeblair: you're patching it up then ? :-)23:28
shashankhegdeanteaya: It still requires a bit of work before I can push it upstream23:28
anteayaSukhdev: thank you23:28
anteayashashankhegde: understood, and thank you23:29
shashankhegdeI'll try to complete it and push it so that everyone can take advantage of it23:29
*** devkulkarni has quit IRC23:29
anteayashashankhegde: thank you23:29
*** signed8bit_ZZZzz is now known as signed8b_23:29
anteayashashankhegde: do you work on jenkins plugins?23:30
*** signed8b_ has quit IRC23:30
*** signed8bit has joined #openstack-infra23:30
*** thedodd has quit IRC23:31
Sukhdevanteaya: shashankhegde and I both work for Arista. He and I put together the whole Arista CI solution -23:31
anteayayour silent partner23:31
anteayayou brought him out of silence23:31
Sukhdevanteaya: yes, silent, but, very powerful partner :-)23:32
anteayathe best kind to have23:32
Sukhdevanteaya: no kidding!!!23:32
anteayaokay I have to get a walk in when it isn't raining and the sun is still out23:33
*** gyee has joined #openstack-infra23:33
anteayaback in a while23:33
*** signed8bit has quit IRC23:34
*** UtahDave has quit IRC23:35
jeblairjhesketh, clarkb: i see the problem23:39
jeblairclarkb: yes, please proceed with removing envinject23:39
clarkbok will do23:41
*** atiwari has joined #openstack-infra23:42
clarkb06 is ready so I am going ahead with it. 05 has one job left23:42
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Fix env var interpretation in zuul swift upload
jeblairclarkb, jhesketh: ^23:42
clarkbjeblair: will that actually fix it though? on the parameters page it looks like the value is truncated?23:43
clarkbjeblair: I agree that is a bug23:43
jeblairclarkb: why do you think it's truncated?23:44
jeblairi mean, it looks like it has multiple values strung together...23:44
clarkbjeblair: because it doesn't show the parameter on multiple lines23:44
clarkband I think when I tested it on jenkins-dev it showed me multiple lines23:44
clarkbI would haveto double check that though23:45
jeblairclarkb: the newlines don't seem to be displayed, but my local testing suggests they are there.23:45
*** oomichi has joined #openstack-infra23:45
clarkbjeblair: ah ok23:45
clarkbanyways +2 on the fix23:45
jeblairclarkb: it's possible the string -> text change may change the display of that on the parameter listing page23:45
clarkbohhhh ya good point23:45
clarkba rendering vs content thing23:46
jeblairclarkb: however, i haven't done that yet -- i just set up the base test case locally and noticed it worked :)23:46
jheskethjeblair: +A since your testing works.. however I'm wondering how it got there in the first place23:46
jheskethie I tested that script :-s23:46
jeblairjhesketh: i actually have not tested that script, i just tested the gearman->jenkins path to make sure the env variable looks correct;  so i'm pretty sure that's the problem23:47
jheskethmaybe I fat-fingered it after the test23:48
*** jcoufal has quit IRC23:48
clarkb06 is done23:48
*** devkulkarni has joined #openstack-infra23:48
*** nati_uen_ has quit IRC23:49
openstackgerritA change was merged to openstack-infra/config: Fix env var interpretation in zuul swift upload
jeblairjhesketh, clarkb: yes, changing to TextParameter causes jenkins to display it in a multiline field23:50
jeblair(well, causes everything to display in a multiline field, which is a little awkward)23:50
jeblairthis may be a good general improvement to gearman-plugin, for folks who would want to see such info, or possibly when interacting with other plugins (like rebuild, which uses saved data)23:50
jeblairmaybe it should scan the value for '\n' and chose string or text appropriately23:51
*** IAX has joined #openstack-infra23:51
jeblair(or just accept the wasted realestate and use text everywhere)23:51
openstackgerritA change was merged to openstack/requirements: Update openstack-doc-tools
*** shayneburgess has quit IRC23:53
*** unicell has quit IRC23:53
clarkbok 05 is ready now23:54
openstackgerritJames E. Blair proposed a change to openstack-infra/gearman-plugin: Use TextParameterValue instead of String
jeblairin case we decide we want that^23:54
jeblairbut yeah, doing a "python -c" command on a parameter with a newline works either way23:55
*** shashankhegde has quit IRC23:55
jheskethjeblair: I don't mind the scan option... it's probably a good idea since it'll allow people to see all the hmac lines if they need23:55
clarkb05 restarted23:55
clarkbputting 04 and 03 in shutdown mode now then probably walking home and picking up from there23:56
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Export prepare_node variables rather than pass them around
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Add path to installing puppet 3 on nodepool nodes
clarkbok thats done23:57
*** prad has quit IRC23:59
*** wenlock has quit IRC23:59
*** msabramo has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!