Thursday, 2013-09-12

clarkbfungi: I think I have decided I really don't like this magic _dir() method in the git-review tests00:00
clarkbimpossible to know what any path actually is00:00
hub_capmordred: i have verified. no black magic.00:00
fungiclarkb: reasonable concern. i would +2 a replacement00:00
hub_capobviously, we will freak out if you put your own backup in its place (checksum validation)00:00
hub_capand try to restore from it00:00
*** mriedem has quit IRC00:01
hub_capbut you can freely take the xtra stuff out00:01
hub_capand put it in your pocket00:01
fungithat's actually non-obvious, but also something we would be unlikely to randomly try anyway00:01
hub_capand fire proof gun safe00:01
*** hashar has quit IRC00:01
mordredhub_cap: that's great.00:01
mordredhub_cap: I think we'll put on the todo list starting to have you do backups00:02
fungi(the putting our backup in place of the rs-provided backup in swift causing an insta-restore from it, i mean)00:02
hub_capfungi: /me hopes00:02
*** hemna is now known as hemnafk00:02
*** Ryan_Lane has quit IRC00:04
*** HenryG_ has joined #openstack-infra00:04
*** HenryG__ has joined #openstack-infra00:08
*** dkehn has quit IRC00:08
*** HenryG__ has quit IRC00:08
*** HenryG has quit IRC00:08
*** dkehn has joined #openstack-infra00:08
*** HenryG_ has quit IRC00:12
*** gyee has quit IRC00:14
*** gyee has joined #openstack-infra00:15
*** prad has joined #openstack-infra00:15
*** zul has quit IRC00:17
*** atiwari has quit IRC00:19
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Update old bug format in recheckwatch
jog0jeblair: hopefully 3rd time is the charm :/00:24
*** prad has quit IRC00:24
*** zul has joined #openstack-infra00:26
*** gyee has quit IRC00:28
* mordred believes in jog000:28
clarkbmordred: fungi
clarkbthough at this point maybe waiting for tomorrow is better? (bup doesn't run for anount ~6 hours so it happens over night anyways)00:41
* fungi has a look00:41
fungii'm cool with now assuming it looks okay00:42
fungi0041 clarkb mordred: fungi
mordredclarkb: looking00:43
fungier, paste fun there. excuse me ;)00:43
mordredclarkb: I +2'd, but did not aprv00:44
*** FallenPegasus has joined #openstack-infra00:44
*** prad has joined #openstack-infra00:44
*** MarkAtwood has quit IRC00:48
*** FallenPegasus has quit IRC00:49
fungilgtm, aprv'd00:50
openstackgerritA change was merged to openstack-infra/config: Use bup to backup review.o.o to an offsite server
*** weshay has quit IRC00:52
fungi(slowly, from the wearable, as gerrit is a bit of a pain on a tiny screen)00:52
*** prad has quit IRC00:52
*** dcramer_ has quit IRC00:56
*** marun has quit IRC01:03
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Delete closed bugs after 5 days in recheckwatch
*** dcramer_ has joined #openstack-infra01:09
clarkbwoo more apparent gerrit differences in the new version01:12
clarkblooks like bouncy castle in the new version will generate its own key and kill your old one01:12
clarkband it does it in a weird format01:12
clarkbmaybe, I am still having trouble making this consistent01:13
jog0jeblair: thanks for the recheck pickle it helped make debugging much easier01:18
*** dkliban has joined #openstack-infra01:19
*** nosnos has joined #openstack-infra01:22
*** svarnau has quit IRC01:25
*** reed has quit IRC01:31
*** dprince has joined #openstack-infra01:36
*** fifieldt has joined #openstack-infra01:38
*** dzyu has joined #openstack-infra01:44
*** melwitt has quit IRC01:51
*** yaguang has joined #openstack-infra01:52
*** HenryG has joined #openstack-infra01:56
*** thomasm has joined #openstack-infra01:59
*** dcramer_ has quit IRC02:02
*** dzyu has quit IRC02:05
*** xchu has joined #openstack-infra02:09
*** dzyu_ has joined #openstack-infra02:10
*** dzyu_ is now known as dzyu02:10
*** dzyu_ has joined #openstack-infra02:14
*** dprince has quit IRC02:15
*** dzyu has quit IRC02:17
*** dzyu_ is now known as dzyu02:17
*** nati_ueno has quit IRC02:22
*** dims has quit IRC02:25
*** xchu has quit IRC02:27
*** jcoufal has quit IRC02:28
*** fifieldt has quit IRC02:29
*** xchu has joined #openstack-infra02:39
*** senk has joined #openstack-infra02:41
*** dzyu_ has joined #openstack-infra02:41
*** jhesketh__ has quit IRC02:41
*** dzyu has quit IRC02:43
*** dzyu_ is now known as dzyu02:43
mordredthat's the best sentence in here ever ^^02:46
* mordred wants a recheck pickle02:46
*** HenryG has quit IRC02:53
*** jhesketh__ has joined #openstack-infra02:54
*** nosnos has quit IRC02:57
*** anteaya__ is now known as anteaya02:57
*** nosnos has joined #openstack-infra02:58
openstackgerritClark Boylan proposed a change to openstack-infra/git-review: Add test cleanups before things they cleanup.
openstackgerritClark Boylan proposed a change to openstack-infra/git-review: Use bouncy castle when testing if available.
clarkbfungi: ^ so many hacks, I am not entirely happy with that but it gives you a general idea of what is needed03:01
*** UtahDave has joined #openstack-infra03:03
openstackgerritClark Boylan proposed a change to openstack-infra/git-review: Use bouncy castle when testing if available.
*** xchu has quit IRC03:12
*** xchu has joined #openstack-infra03:30
*** anteaya has quit IRC03:33
*** hashar has joined #openstack-infra03:36
hub_capmordred: a recheck pickle sounds like a dreaded followup appt w/ your doctor03:37
*** FallenPegasus has joined #openstack-infra03:45
*** vogxn has joined #openstack-infra03:46
*** reed has joined #openstack-infra03:53
*** fifieldt has joined #openstack-infra03:55
*** llu has joined #openstack-infra03:57
*** dcramer_ has joined #openstack-infra03:58
lluIt seems the openstack-requirements Jenkin's job keeps failing,
llupip seems complaining about duplicated requirement of hacking in both global-requirements and test-requirements04:05
lluwhich file should we remove the hacking requirement from? global or test?04:05
*** jhesketh__ has quit IRC04:07
*** jhesketh has joined #openstack-infra04:07
*** senk has quit IRC04:11
*** dcramer_ has quit IRC04:11
clarkbllu: neither04:11
clarkbllu: is the fix04:12
*** SergeyLukjanov has joined #openstack-infra04:13
*** SergeyLukjanov has quit IRC04:14
*** SergeyLukjanov has joined #openstack-infra04:20
*** FallenPegasus has quit IRC04:21
*** dcramer_ has joined #openstack-infra04:24
*** SergeyLukjanov has quit IRC04:30
*** hashar has quit IRC04:32
*** SergeyLukjanov has joined #openstack-infra04:33
*** dguitarbite has quit IRC04:33
*** dguitarbite has joined #openstack-infra04:35
*** FallenPegasus has joined #openstack-infra04:37
openstackgerritTom Fifield proposed a change to openstack-infra/config: adding staging server for activity board
*** dcramer_ has quit IRC04:41
*** senk has joined #openstack-infra04:42
*** FallenPegasus has quit IRC04:43
*** senk has quit IRC04:46
*** dmakogon_ has joined #openstack-infra04:48
*** SergeyLukjanov has quit IRC04:48
*** pcrews has joined #openstack-infra04:50
*** Ryan_Lane has joined #openstack-infra04:52
*** SergeyLukjanov has joined #openstack-infra04:57
*** SergeyLukjanov has quit IRC04:57
*** vogxn has quit IRC05:00
*** vogxn has joined #openstack-infra05:02
*** mrodden has joined #openstack-infra05:05
*** reed has quit IRC05:11
*** dguitarbite has quit IRC05:13
*** senk has joined #openstack-infra05:14
*** llu has left #openstack-infra05:15
*** dguitarbite has joined #openstack-infra05:16
*** pmoosh has quit IRC05:18
*** senk has quit IRC05:18
*** dkliban has quit IRC05:21
*** boris-42 has joined #openstack-infra05:22
*** dguitarbite has quit IRC05:22
*** dguitarbite has joined #openstack-infra05:23
*** nicedice has quit IRC05:29
*** dguitarbite has quit IRC05:30
*** harlowja has quit IRC05:31
*** harlowja has joined #openstack-infra05:31
*** harlowja_ has joined #openstack-infra05:36
*** harlowja has quit IRC05:38
*** harlowja has joined #openstack-infra05:39
*** harlowja_ has quit IRC05:42
*** SergeyLukjanov has joined #openstack-infra05:42
*** pblaho has joined #openstack-infra05:42
*** UtahDave has quit IRC06:01
*** nosnos has quit IRC06:17
*** nosnos has joined #openstack-infra06:24
*** yolanda has joined #openstack-infra06:29
*** flaper87|afk is now known as flaper8706:33
*** yolanda has quit IRC06:37
*** vogxn has quit IRC06:39
*** vogxn has joined #openstack-infra06:41
*** yongli_away has quit IRC06:43
*** SergeyLukjanov has quit IRC06:43
*** amotoki has joined #openstack-infra06:52
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Publish Over SSH Plugin option added
*** xchu has quit IRC06:56
*** afazekas has joined #openstack-infra06:57
*** pcrews has quit IRC06:58
*** flaper87 is now known as flaper87|afk07:05
*** afazekas_ has joined #openstack-infra07:05
*** afazekas_ has quit IRC07:06
*** dachary has quit IRC07:06
*** dachary has joined #openstack-infra07:06
*** flaper87|afk is now known as flaper8707:14
*** shardy_afk is now known as shardy07:15
*** xchu has joined #openstack-infra07:15
*** vogxn has quit IRC07:15
*** vogxn has joined #openstack-infra07:16
*** Ryan_Lane has quit IRC07:21
*** xchu has quit IRC07:26
*** vogxn has quit IRC07:28
*** dmakogon_ has quit IRC07:28
*** jpich has joined #openstack-infra07:41
*** xchu has joined #openstack-infra07:44
*** flaper87 is now known as flaper87|afk07:47
*** Bada has joined #openstack-infra07:48
*** afazekas has quit IRC07:55
*** boris-42 has quit IRC07:55
*** flaper87|afk is now known as flaper8707:58
*** xchu has quit IRC08:08
*** SergeyLukjanov has joined #openstack-infra08:09
*** yolanda has joined #openstack-infra08:10
*** fbo_away is now known as fbo08:17
*** locke105 has quit IRC08:23
*** bashok has joined #openstack-infra08:50
*** kspear has quit IRC09:04
*** zaro has quit IRC09:05
*** zaro has joined #openstack-infra09:06
*** fifieldt has quit IRC09:13
*** salv-orlando has joined #openstack-infra09:20
*** boris-42 has joined #openstack-infra09:22
*** dzyu has quit IRC09:35
*** xchu has joined #openstack-infra09:51
*** odyssey4me has joined #openstack-infra09:57
*** Bada has quit IRC10:03
*** xchu has quit IRC10:08
*** branen_ has quit IRC10:21
*** yaguang has quit IRC10:26
*** mdenny has quit IRC10:26
*** dims has joined #openstack-infra10:29
*** SergeyLukjanov has quit IRC10:36
*** SergeyLukjanov has joined #openstack-infra10:38
*** afazekas has joined #openstack-infra10:51
*** SergeyLukjanov has quit IRC10:54
*** kspear has joined #openstack-infra11:00
*** weshay has joined #openstack-infra11:02
*** dizquierdo has joined #openstack-infra11:03
*** dprince has joined #openstack-infra11:05
*** SergeyLukjanov has joined #openstack-infra11:07
*** DennyZhang has joined #openstack-infra11:27
*** pcm_ has joined #openstack-infra11:32
*** jhesketh has quit IRC11:40
*** HenryG has joined #openstack-infra11:40
*** ArxCruz has joined #openstack-infra11:43
*** chuckieb has joined #openstack-infra11:44
*** rfolco has joined #openstack-infra11:45
*** kspear has quit IRC11:51
*** kspear has joined #openstack-infra11:52
*** zeus has joined #openstack-infra11:59
*** jcoufal has joined #openstack-infra12:03
*** jcoufal has quit IRC12:03
openstackgerritSergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Build Blocker Plugin support added
*** jcoufal has joined #openstack-infra12:04
*** jcoufal has quit IRC12:04
*** mgagne has quit IRC12:05
*** mgagne has joined #openstack-infra12:05
*** mgagne has joined #openstack-infra12:05
*** jcoufal has joined #openstack-infra12:06
*** sld has quit IRC12:07
*** annegentle has quit IRC12:08
*** sld has joined #openstack-infra12:09
*** dprince has quit IRC12:09
*** alexpilotti has joined #openstack-infra12:18
*** flaper87 is now known as flaper87|afk12:23
*** changbl has quit IRC12:27
*** dizquierdo has quit IRC12:28
*** flaper87|afk is now known as flaper8712:31
*** DennyZhang has quit IRC12:34
*** afazekas_ has joined #openstack-infra12:39
*** KLevenstein has joined #openstack-infra12:47
*** adalbas has joined #openstack-infra12:48
*** afazekas_ has quit IRC12:49
*** nhm has quit IRC12:50
*** anteaya has joined #openstack-infra12:56
*** annegentle has joined #openstack-infra12:57
*** annegentle is now known as annegentle_away13:02
*** sandywalsh has quit IRC13:08
*** matty_dubs|gone is now known as matty_dubs13:10
*** mriedem has joined #openstack-infra13:15
*** zul has quit IRC13:18
*** zul has joined #openstack-infra13:18
*** nosnos has quit IRC13:26
*** dprince has joined #openstack-infra13:27
*** lnxnut has joined #openstack-infra13:29
sdaguehub_cap: I'll try to look today, it's been a kind of crazy week13:32
*** ssafron has joined #openstack-infra13:32
ssafronIs it possible to delete an Etherpad from  I accidentally put some sensitive information on one, and now it looks like I cannot get rid of the history.13:36
anteayassafron: when fungi is around perhaps you can send him an email with the link to the etherpad in question13:37
anteayaor pm him (your choice)13:37
*** changbl has joined #openstack-infra13:38
ssafronOK thanks anteaya13:38
anteayaI hope he can do something to help13:38
sdaguessafron: the way etherpads do versioning this isn't very easy13:38
sdagueI think the only option is to nuke the whole thing, but that's not really an option if it's one community members have been using13:39
*** dkranz has joined #openstack-infra13:39
ssafronNope, this was a new etherpad just used by two people one day.  So I'd be fine if it just gets totally deleted.13:40
fungissafron: i'll have to take a look at the db schema. if it's a fairly safe and simple set of delete queries then i'm happy to redact it for you13:43
fungissafron: just /msg me the url13:43
*** markmcclain has quit IRC13:45
*** lnxnut has quit IRC13:50
*** burt has joined #openstack-infra13:53
*** alexpilotti has quit IRC13:53
*** alexpilotti has joined #openstack-infra13:56
*** safron has joined #openstack-infra14:00
*** ssafron has quit IRC14:00
*** safron has quit IRC14:02
*** safron has joined #openstack-infra14:02
*** dizquierdo has joined #openstack-infra14:03
*** vogxn has joined #openstack-infra14:04
anteayafungi from what I am reading puppet-dashboard needs a puppet-master to talk to, the master sends requests to the dashboard14:05
anteayaI am not seeing how to get a puppet agent talking to the dashboard14:06
anteayais this consistent with your understanding of the puppet-dashboard?14:06
*** markmcclain has joined #openstack-infra14:07
fungianteaya: what we have in /etc/puppet/puppet.conf on our servers includes a line for it in the [master] section...14:07
*** alexpilotti has quit IRC14:07
anteayaah ha, thanks14:07
fungii believe that configures each server to report directly to the dashboard server14:08
anteayaeach server, meaning each puppet-master?14:08
fungino, each puppeted machine14:08
fungieach server running an agent14:09
fungi(i snarfed that example directly from but it's on all of them)14:09
anteayareally, okay I will see if inserting that line into puppet.conf on the agent I am running gives me any results14:09
anteayaokay thanks14:10
fungiin the [master] section we also have:14:11
fungireports=store, http14:11
fungiand then in the [agent] section:14:11
fungii'm running tcpdump on a server right now to see if it's actually trying to connect directly to to submit reports, just as a sanity check14:12
fungimight take up to 10 minutes to know for sure14:12
anteayathank you14:14
*** dkliban has joined #openstack-infra14:15
*** alexpilotti has joined #openstack-infra14:15
*** pabelanger_ has joined #openstack-infra14:16
*** pabelanger__ has joined #openstack-infra14:16
*** pabelanger__ has quit IRC14:17
*** pabelanger__ has joined #openstack-infra14:17
*** pabelanger has quit IRC14:17
*** pabelanger__ is now known as pabelanger14:17
*** pabelanger_ has quit IRC14:17
*** pabelanger_ has joined #openstack-infra14:17
*** pabelanger has quit IRC14:17
*** pabelanger has joined #openstack-infra14:18
*** dcramer_ has joined #openstack-infra14:19
*** mrodden has quit IRC14:24
*** annegentle_away is now known as annegentle14:24
*** adalbas has quit IRC14:28
*** dkliban has quit IRC14:30
*** afazekas has quit IRC14:30
fungimmm, you may be right. i don't see the server trying to open a socket to the dashboard to report anything. lemme make sure the agent is running and not stuck14:30
*** senk1 has joined #openstack-infra14:33
sldis there anyone around that can answer a quick jenkins job question?14:33
Mithrandirthat depends on what the question is14:33
Mithrandirdon't ask to ask14:34
fungisld: yeah, just ask and we'll answer if we know ;)14:34
*** mrodden has joined #openstack-infra14:34
sdaguefungi: the db schema for etherpad is "interesting" :)14:35
sldlooking to clean remote ftp directory, and then put newly sphinx-generated content there...the problem is this -- after the clean remote is in place, and while the ftp upload is in process, there is no site content..  are there any other suggestions for ways to accomplish the same thing, but maybe done a different way?14:35
fungisdague: yes, i ended up using the api, which worked (somewhat to my surprise)14:35
*** atiwari has joined #openstack-infra14:36
sdaguefungi: nice, you got an example? I think I only managed to bork up things the last time I tried to do that14:36
Mithrandirsld: rsync with --delete-after?14:36
fungisld: so that's more of an ftp publisher plugin question i guess. for openstack, we don't use the clean remote option... we just dump new files on top of old ones and accept (currently) that cruft will pile up14:36
sldfungi: exactly - that is what i am trying to solve for... cleaning that. ;)14:37
fungibut yes, something like what Mithrandir suggests would be cleaner. no idea of jenkins has an rsync pubisher plugin or if you'd have to hack something up for it14:37
*** senk1 has quit IRC14:37
sldi didn't see any rsync plugin on the list, but if i need to make a new one, that's fine too.14:37
*** pentameter has joined #openstack-infra14:37
slderr rsync publisher, rather.14:37
Mithrandiryou could also use the ftp publisher and have something that rm's based on ctime.14:38
*** senk1 has joined #openstack-infra14:38
fungiin our case, we're using ftp with a somewhat braindead web hosting service which we can't install arbitrary binaries ike rsync onto, so we haven't really played around with it. we do have jobs which generate content and then rsync them to an arbitrary server via ssh however14:38
fungiand that works fairly well14:38
sldif there is any way to pass options to rsync, ...that might be good. ;-)14:39
fungisdague: yes, writing something up to add to our server documentation here in a bit14:39
sldi'll scour source for an rsync example... thanks. :)14:39
Mithrandiror you could have a symlink that you make sure sorts last and which is then flipped afterwards.14:39
Mithrandirso you publish to foo-$jobid and have a foo/ symlink pointing to the right place14:39
fungisld: yes, in our case the job is parameterized and passes things into a shell script which ends with the rsync to put the results into place, rather than relying on an explicit jenkins publisher14:40
anteayafungi k14:40
jeblairfungi: that requires a level of access that i don't think we want to give the builders14:40
fungijeblair: right, for our doc jobs i don't think it's a good idea14:41
jeblairfungi: i think sld is talking about our jobs :)14:41
sldfungi: can you think of a job that does that, that i can look at as an example?14:41
fungii totally misread and thought he was talking about another jenkins14:41
Mithrandirjeblair: not really, you can use command= in the authorized_keys file or one of the rsynconly scripts.14:41
*** vogxn has quit IRC14:42
*** SergeyLukjanov has quit IRC14:42
jeblairMithrandir: yes, that's possible once we restructure where we publish docs14:43
Mithrandirwell, that's something I have no idea how's done today. :-)14:43
Mithrandirjust saying you don't need to give out extra perms to the builders.14:43
annegentlejeblair: one idea we're thinking of is docs-draft for everything published from master14:43
jeblairMithrandir: two things need to change: 1) use a site we can ssh to instead of ftp, and 2) generate special purpose keys, so the same key isn't used for publishing logs or tarballs as docs14:43
*** senk has joined #openstack-infra14:43
annegentlejeblair: sld is working through it but wanted to throw that out there14:44
jeblairMithrandir: but yeah, if we did that, we could use command=14:44
Mithrandir(I'd also like a central-publish plugin that grabs the bits to be published, sends them to the master and the master does the publishing, so you don't need to let arbitrary slaves log in lots of places)14:44
annegentledoh afk for a bit14:44
*** adalbas has joined #openstack-infra14:44
jeblairMithrandir: well, that's what happens with most publishers, so that's the behavior that ftp, scp, and ssh publishers have now14:44
*** senk2 has joined #openstack-infra14:44
*** pblaho has quit IRC14:45
Mithrandirjeblair: oh, ok.  It used not to be that way, so if that's fixed, that's great. :-)14:45
fungii suppose one workaround to not letting arbitrary build slaves rsync to the (eventual not rackspace cloudsites) webserver would be to generate and archive the content, then have a follow-on job which retrieves it on a trusted slave and does the rsync14:46
jeblairMithrandir: i just double checked -- the ssh logins for the scp publishing plugin only come from our two masters14:46
fungiwhich i guess is more or less what Mithrandir just suggested as well14:46
jeblairMithrandir: i'm assuming the others work similarly14:46
*** senk has quit IRC14:48
*** senk1 has quit IRC14:48
jeblairon further thought, there's one other 'security' aspect that we get from the current config, and that is that the destination path is configured as part of the job14:48
Mithrandiryou get that with the rsynconly shell things too14:49
jeblairif the 'docs' ssh key were available inside the job (and not just to jenkins) then it means that any docs job could alter all docs sites, whereas currently, the worst it can do is alter its subset (eg, nova)14:49
*** CaptTofu has quit IRC14:50
Mithrandirthen have one ssh key per job?14:50
jeblairthe artifact storage system i wrote about in an email to openstack-infra could probably solve this problem too.14:51
*** senk2 has quit IRC14:51
*** CaptTofu has joined #openstack-infra14:51
*** afazekas has joined #openstack-infra14:51
jeblair(since it's designed to scope things by job)14:51
sldwhen was it?  or do you have a link handy?14:52
jeblairMithrandir: yes, i think that would work.14:52
*** senk has joined #openstack-infra14:53
*** rnirmal has joined #openstack-infra14:54
sldthx...i'll finish reading it tonight when i get back to the office...headed home and to bed now. :)14:55
*** senk1 has joined #openstack-infra14:56
anteayafungi: was the puppet agent stuck?14:56
*** sld is now known as sld-away14:56
fungianteaya: no, i got sidetracked when i discovered we have servers where puppet is complaining that the pip provider is broken14:57
anteayaah okay14:57
*** DennyZhang has joined #openstack-infra14:58
fungicurrent suspicion is that a recent non-distro-package upgrade of pip has moved the binary to a path where puppet doesn't know to look for it14:58
fungitrying to nail this down real quick14:58
*** senk has quit IRC14:58
dhellmann_good morning everyone15:00
*** dhellmann_ is now known as dhellmann15:00
anteayagood morning dhellmann15:00
dhellmannthings seem quiet, so I'm hoping that means people have the bandwidth to review today :-)15:01
*** mrodden1 has joined #openstack-infra15:02
*** thedodd has joined #openstack-infra15:02
*** mrodden has quit IRC15:02
fungiha, apparently lifeless already tripped over this issue a couple weeks ago... bug 121869615:03
uvirtbotLaunchpad bug 1218696 in openstack-ci "new puppetmaster ends up with broken pip provider" [Undecided,New]
*** senk1 has quit IRC15:03
fungithough it's not necessarily just puppet master servers affected. i see it on our gerrit servers15:04
anteayaI'll just go ahead and work on installing a puppet-master to talk to the dashboard15:06
anteayaand if you get a chance later, I would be interested in your tcpdump findings regarding puppet agent/dashboard communication or lack thereof15:07
anteayaI'm referring to my little test infra15:07
*** dkliban has joined #openstack-infra15:11
fungijeblair: did you mean to leave puppet stopped on zuul yesterday?15:14
jeblairfungi: i did not; oops.15:15
fungijust making sure15:15
jeblairfungi: it should be safe to restart now if you want15:15
*** pcrews has joined #openstack-infra15:15
*** senk1 has joined #openstack-infra15:16
*** dcramer_ has quit IRC15:17
*** odyssey4me has quit IRC15:17
*** mrodden has joined #openstack-infra15:18
*** senk has joined #openstack-infra15:19
*** pycabrera has joined #openstack-infra15:19
*** senk1 has quit IRC15:21
*** SergeyLukjanov has joined #openstack-infra15:21
*** mrodden1 has quit IRC15:23
*** mrodden1 has joined #openstack-infra15:26
*** mrodden has quit IRC15:26
*** UtahDave has joined #openstack-infra15:27
*** prad has joined #openstack-infra15:28
*** dcramer_ has joined #openstack-infra15:29
fungianteaya: yeah, trying a few systems, i don't see any of the agents trying to report to the dashboard directly, only the master server seems to want to report to the dashboard15:32
ekarlso-would it be easy to make JJB accept multiple directories?15:34
*** markmcclain has quit IRC15:35
*** bashok has quit IRC15:36
*** rnirmal_ has joined #openstack-infra15:39
anteayafungi: thanks, your findings are consistent with the puppetlabs documentation15:40
*** branen has joined #openstack-infra15:40
anteayaso my assumption that I could get an agent talking to the dashboard is wrong15:40
hub_capsdague: hey i got a +2 from dtroyer, can you look @ trove+devstack
*** rnirmal has quit IRC15:40
*** rnirmal_ is now known as rnirmal15:40
hub_capthen i can say im making progress to the infra guys when they ask about trove gating tests!!15:40
*** pabelanger has quit IRC15:44
*** rnirmal has quit IRC15:50
*** rnirmal has joined #openstack-infra15:51
*** KennethWilke has joined #openstack-infra15:51
fungiokay, so restarting the puppet agent daemon solves the dilemma of the moving pip executable. bug 1218696 updated with relevant info15:53
uvirtbotLaunchpad bug 1218696 in openstack-ci "new puppetmaster ends up with broken pip provider" [Low,Triaged]
*** mgagne has quit IRC15:54
pleia2anteaya: doing some headache triage away from the screen for a bit, hopefully will be back in not too long :)15:54
anteayalet me see what I can do with the notes you left me15:55
pleia2ok, great15:55
anteayatake care of your headache15:55
*** pabelanger_ has quit IRC15:55
*** pabelanger has joined #openstack-infra15:56
*** svarnau has joined #openstack-infra15:59
*** mgagne has joined #openstack-infra16:00
*** mgagne has joined #openstack-infra16:00
*** lnxnut_ has joined #openstack-infra16:01
*** boris-42_ has joined #openstack-infra16:02
*** marun has joined #openstack-infra16:02
*** boris-42_ has quit IRC16:04
*** mgagne1 has joined #openstack-infra16:04
*** mgagne1 has joined #openstack-infra16:04
*** boris-42 has quit IRC16:06
*** lnxnut_ has quit IRC16:06
*** mgagne has quit IRC16:06
*** senk has quit IRC16:08
*** thedodd has quit IRC16:10
*** fbo is now known as fbo_away16:11
*** mgagne1 has quit IRC16:13
*** changbl has quit IRC16:14
*** dcramer_ has quit IRC16:16
*** fbo_away is now known as fbo16:24
*** rnirmal has quit IRC16:25
*** rnirmal has joined #openstack-infra16:25
*** dims has quit IRC16:25
*** mgagne has joined #openstack-infra16:26
*** mgagne has joined #openstack-infra16:26
UtahDavefungi: did you get a chance to test a Salt Jenkins slave?16:27
fungiUtahDave: i'm currently playing with determining why we're not autoupgrading our salt packages to the available later version...
fungiUtahDave: don't want to revert the change where we disabled it until i can confirm we're properly upgrading16:29
fungineed to spot check a few more systems still16:29
*** markmcclain has joined #openstack-infra16:30
fungilooks like it's upgrading correctly on our centos6 slaves but not on precise16:31
*** moted has quit IRC16:31
*** moted has joined #openstack-infra16:32
fungii think possibly our auto package updates may not upgrade all packages, but instead only from certain package repositories. i'll test that out16:32
*** danger_fo_away is now known as danger_fo16:34
fungiahh yes, we need to add an entry to Unattended-Upgrade::Allowed-Origins16:34
*** dcramer_ has joined #openstack-infra16:34
*** fbo is now known as fbo_away16:36
*** dims has joined #openstack-infra16:37
jeblairclarkb: is it correct to say that the swift hpcloud issue is that they don't recognize directory markers when generating indexes?16:39
UtahDavefungi: ok, good.16:39
fungiassuming this test works as intended, i'll upload the auto-updates fix, the one to reenable salt, one to make a small adjustment to the launch instructions for the new --salt option and the one to puppetize the special-purpose trigger slave as a stack of reviews16:39
jeblairclarkb: or they don't support setting directory markers at all, or ...?16:39
clarkbjeblair: it is a little more subtle than that16:40
clarkbjeblair: they seem to do the correct thing for your private view of containers, but not in the public view16:40
clarkbI assume to reduce load as private is presumably going to be accessed less often?16:40
*** reed has joined #openstack-infra16:40
fungipresumably similar to why rackspace only allows accessing public swift object urls via cdn16:41
UtahDavefungi: that sounds great. Will I need to make any changes to my review? I think mine still is set to work on all jenkins slaves.16:41
jeblairclarkb: ok, so is 'they don't recognize directory markers when generating index pages for the public view of containers' accurate enough to commit to a mailing list archive? :)16:41
clarkbjeblair: I think so, it is possible that they do support it afterall but I couldn't find documentation to that effect16:41
fungiUtahDave: yeah, we'll rebase yours on top of the change which adds that slavce16:41
fungiand make some minor tweaks to have it be specific to that host16:42
UtahDavefungi: great16:42
notmynameclarkb: jeblair: "when generating index pages" as in swift's staticweb feature?16:42
clarkbnotmyname: aye16:42
notmynameclarkb: ah, so perhaps we have a bug in staticweb?16:42
clarkbnotmyname: or pebkac16:42
clarkbwith me being between chair and keyboard16:43
fungiclark exists between chair and keyboard16:43
*** matty_dubs is now known as matty_dubs|lunch16:44
clarkbjeblair: notmyname: let me give it another shot right now16:45
*** clayg has joined #openstack-infra16:46
claygIf I'm diggin' on a failed gate job logs, is there anyway to get back to the jenkins job-id?16:47
clayglike I have - but I want
*** dcramer_ has quit IRC16:49
jeblairnotmyname: FYI  (we're trying very hard to store build artifacts in swift)16:51
jeblairclayg: why?  what's missing from the logs.o.o site?16:51
claygjeblair: just curious I suppose?16:52
jeblairclayg: (i am thinking about your question; it's difficult)16:53
claygjeblair: I mean I *am* sorta at the moment sorta wondering about "Other changes tested concurrently with this change"16:54
jeblairclayg: I think it's not possible to do directly; you'd have to hit each of the jenkins masters and look at all the jobs from around that time and examine the parameters16:54
*** mgagne has quit IRC16:54
jeblairclayg: we could probably add some output to that to make it easier to trace back16:54
claygjeblair: so but surely the... :D16:54
jeblairclayg: we can also add 'Other changes tested concurrently with this change' output to the console log as well16:55
jeblairclayg: the reason why we have not made it easy to go in the direction you're asking about...16:55
jeblairclayg: is that build logs are deleted from jenkins fairly quickly (i think they stay around for 24 hours)16:55
jeblairclayg: logs.o.o logs stay around for 6 months16:55
*** mgagne has joined #openstack-infra16:57
*** mgagne has joined #openstack-infra16:57
jeblairclayg: you can fetch the zuul ref and get exactly what was tested for each project16:57
jeblairclayg: (and if you look at git log, you should see where zuul merged the other changes)16:57
jeblairclayg: look at
jeblairclayg: and looke for lines like: git fetch refs/zuul/master/Z3b4fde5321bf4207889978b69131ae8716:58
*** ruhe has joined #openstack-infra16:58
jeblairclayg: so you can always use that to locally checkout and examine exactly what was tested16:58
claygcool - didn't know about that - helpful16:59
*** mdenny has joined #openstack-infra16:59
*** Ryan_Lane has joined #openstack-infra17:00
*** hashar has joined #openstack-infra17:01
*** senk has joined #openstack-infra17:01
*** senk has quit IRC17:02
*** DennyZhang has quit IRC17:02
clarkbjeblair: so some pebkac apparently but it isn't perfect17:03
*** dcramer_ has joined #openstack-infra17:03
clarkbjeblair: notice that dir names are listed twice, once as an actual object and again rendered as a dir17:03
jeblairnotmyname: ^17:03
jeblairclarkb: what's 89381323713085 ?17:04
clarkbjeblair: magical public url for my tenant+project I think17:04
*** thedodd has joined #openstack-infra17:04
clarkbI am going to disable the rendering to see if foo/ and foo are distinct objects17:05
*** jpich has quit IRC17:05
*** Bada has joined #openstack-infra17:05
clarkbapparently setting web-listings to false isn't the correct way to do that17:06
*** dcramer_ has quit IRC17:06
claygclarkb: is this staticweb?17:06
clarkbclayg: yes17:07
*** gyee has joined #openstack-infra17:07
*** nati_ueno has joined #openstack-infra17:07
clarkb maybe foo and testing are not needed?17:08
claygclarkb: the explicity directory markers aren't needed unless you need empty directories17:08
claygnot sure why static web isn't parsing them out and rolling them up - that may be a bug17:08
clarkbI think jclouds creates those because other object stores want them17:08
clayghrmmm... might be static web wants a specific content type17:09
*** pcrews has quit IRC17:10
claygX-Container-Meta-Web-Directory-Type - default is application/directory?17:10
clarkbclayg: no clue, how do I check?17:11
claygswift stat jenkinstesting2 foo17:11
*** nati_ueno has quit IRC17:12
claygsoo that looks right :P17:13
*** ruhe has quit IRC17:13
clarkbclayg: it is also possible this is a bug in whatever version of swift hp has deployed17:13
clarkband may not be a problem in current swift17:13
jeblairha we're talking about swift here and infra in #openstack-swift17:13
jeblairoh!  you only need directory markers for empty directories?  i thought you needed them in order to 'list objects in this pseudo-directory'?17:15
clarkbjeblair: apparently. I can delete foo and see what happens17:15
clarkbfoo deleted, that seems to make it happy17:16
clarkb(note jclouds would need an update if we want to use it this way)17:16
claygclarkb: jeblair: looking at master I dont' see any code to sqeueeze them together - no i think staticweb needs a fix17:17
*** pmoosh has joined #openstack-infra17:17
clarkbclayg: cool I found a bug maybe17:19
*** zehicle has quit IRC17:20
*** rockyg has joined #openstack-infra17:21
claygclarkb: I'm trying out a fix now - it's possible gholt is the only one that really *knows* how staticweb works17:21
claygthe rest of us just sorta poke at it with sticks17:21
*** Ryan_Lane has quit IRC17:23
*** Ryan_Lane has joined #openstack-infra17:23
*** Ryan_Lane has quit IRC17:23
*** Ryan_Lane has joined #openstack-infra17:23
*** zehicle has joined #openstack-infra17:24
*** markmcclain has quit IRC17:27
*** markmcclain has joined #openstack-infra17:28
*** vipul is now known as vipul-away17:30
*** vipul-away is now known as vipul17:31
*** pcm__ has joined #openstack-infra17:31
clarkbfungi: for the pip stuff you ran into, my observation was that once pip upgraded itself, puppet needed to be run again. I thought it was per catalog run and not per process17:32
*** svarnau has quit IRC17:32
fungiclarkb: on our unit test slaves where we run it from cron, yes17:33
fungiclarkb: on systems where the agent is running persistently as a daemon, that needs restarting17:33
clarkbI see. That is a bit annoying17:33
fungiyes, isn't it though17:33
*** HenryG_ has joined #openstack-infra17:33
clarkbpuppet probably shouldn't make config assumptions for things between runs17:34
*** pcm_ has quit IRC17:34
clarkbjeblair: so for hpcloud I think we can probably make this work, we would need to deal with the public url location (proxy? or cave and use CDN and potentially use rackspace and hp?)17:35
*** dcramer_ has joined #openstack-infra17:35
*** HenryG has quit IRC17:36
*** svarnau has joined #openstack-infra17:36
clarkbfungi: was review.o.o one of the hosts with pip puppet trouble? trying to sort out if I should attempt a bup join on that host17:38
*** dizquierdo has quit IRC17:39
claygclarkb: so listings don't really handle directory markers all that great :\17:39
fungiclarkb: it was. i only restarted puppet on it and review-dev, so there may be others out there still showing symptoms17:39
claygclarkb: this gets rid of the dupes ->
*** dcramer_ has quit IRC17:40
claygbut I really should make it a proper fix and promote the application/directory objects to be treated more like real subdir listing entries17:40
clarkbfungi: ok, so bup join tomorrow. thanks17:40
fungiclayg: also, it wasn't so much a config assumption... if you look at the ruby implementing the pip package provider, it just uses the unqualified command name but presumably the path to that ends up being remembered for subsequent invocations17:41
claygat one time I wanted to just teach the container servers how to do it so up stream applications don't all have to independetly solve empty directories themslves17:41
fungier, clarkb not clayg17:41
claygmy mere presense is causing problems17:41
* fungi curses his own lazy nick tab completion habits17:41
*** SergeyLukjanov has quit IRC17:42
*** thomasm has quit IRC17:42
*** Bada has quit IRC17:42
clarkbclayg: while you are here and have looked at that code. Any idea what performance for weblisting would be like when a container has hundreds of thousands of objects?17:42
*** markmcclain has quit IRC17:43
clarkb(maybe millions)17:43
*** Bada has joined #openstack-infra17:43
clarkbjeblair: ^ is another thing to consider, by maintaining an external index we can cache a lot of the info rather than reprocessing17:43
jeblairclayg: i think on balance you're solving more problems than causing, it's nice to have you around :)17:43
*** UtahDave has quit IRC17:43
jeblairclarkb: yeah, that's why i was initially thinking we'd have a shadow db; but if we use formpost, then that's not so easy17:44
*** SergeyLukjanov has joined #openstack-infra17:44
jeblairclarkb: btw, have you caught up on the email thread?17:44
clarkbjeblair: mostly I think17:44
claygclarkb: the container does most of the work (of doing the sudeo nested) - and millions is ok, but not super, hundreds of thousands is probably fine17:45
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add saltstack APT repository to automatic upgrades
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Revert "Disable salt master and minions globally"
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add a Jenkins slave for triggering salt commands
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Mention the --salt option in launch instructions
clarkbjeblair: you are talking about making jobs only able to upload to one place17:45
mrodden1so what version of distribute/setuptools should i be on... i can't "python develop" anything anymore17:46
clarkbjeblair: and possibly having zuul manage that with the swift formpost undocumetned thing17:46
mrodden1and pbr complains that a bunch of build targets are missing17:46
clarkbmrodden1: setuptools > 0.717:46
mrodden1UserWarning: Unknown distribution option: 'install_requires'17:46
mrodden1is the error17:46
clayg... well acctually for many quanties of "millions" millions is probably not fine, it's bad.  But for < a few million we're back into "not super"17:46
mrodden1clarkb: i have 1.1 i believe17:46
clarkbmrodden1: I wonder if you are running into the thing clayg found with setuptools lazy loading on top of distutils17:47
clarkbmrodden1: can you paste a complete traceback?17:47
mrodden1clarkb: sure17:47
jeblairclayg: any individual pseudo-dir would only have up to thousands of entries; but with thousands of those, the whole container would have millions17:47
jeblairclayg: ^ just to be more explicit about the load clarkb is asking about...17:48
jeblairclayg: does that change anything?17:48
claygyeah... I mean many millions of objects in the container will make listings and adding lots of new objects at once sorta... slower than you want17:50
*** rfolco has quit IRC17:50
clarkbya so we will probably need container rollover of some sort17:51
claygjeblair: clarkb: with the cardinality you're thinking of having hundreds of thousands of x-web-listing containers with hundreds of thousands of objects would probably be a better plan17:51
jeblairclarkb: deja-vu!  container rollover with the cdn is hard and the cdn is needed for staticweb on rackspace!17:51
*** dcramer_ has joined #openstack-infra17:51
*** thomasbiege has joined #openstack-infra17:51
clarkbjeblair: right!17:51
fungithis definitely sounds... familiar17:51
* clayg doesn't have thing hung of the term "container rollover"17:52
* clayg raises KeyError17:52
jeblairclayg: i think you and clark are saying similar things -- adding more containers makes this easier17:52
jeblairclayg: except that on rackspace, you need to use the cdn in order to use staticweb17:52
jeblairclayg: and the cdn is a random hostname per-container17:53
*** boris-42 has joined #openstack-infra17:53
jeblairclayg: which means that adding a new container means adding dns entries to each time17:53
clarkbright every time we try to solve this completely with swift I end up thinking swift wasn't built for this17:53
clarkbswift is meant to back an application17:53
jeblairclayg: which brings dns ops into it17:53
clarkbif we write an application that manages all of this stuff it is fine, but there are a few moving pieces to consider17:53
jeblairi tried to suggest we write a swift application, but ttx and jhesketh have not received that idea very well.17:54
jeblairswift-backed application17:54
clarkbjeblair: ya17:54
notmynamejeblair: clayg: actually, the CDN URLs at RAX are deterministic, IIRC. you should be able to calculate them up front17:54
clarkbnotmyname: they looked like a uuid17:54
claygnotmyname: they didn't used to be, but it's still a dns entry17:55
fungithe dns piece would be less worrisome with an actual openstack answer to dns management, but we likely don't want to implement automation based on rackspace's proprietary dns api17:55
notmynameclarkb: clayg: let me check...17:55
jeblairfungi: definitely not my first choice.  also, it's worth taking a step back and noting "constantly changing dns entries" is not a design goal for us, but actually feels like a very hacky workaround.17:56
fungithat too, for sure17:56
*** dcramer_ has quit IRC17:57
*** markmcclain1 has joined #openstack-infra17:57
notmynameclarkb: clayg: looks like an HMAC with a secret component from the config, so you prob can't precompute it
claygnotmyname: CF doesn't use domain remap at all huh?  wildcard dns is out?17:58
clarkbjeblair: the HP non CDN url is at least deterministic once you know the root17:58
clarkbjeblair: not sure about the CDN urls17:58
jeblairclarkb: do you know if hpcloud support formpost?17:59
notmynameclayg: they don't use domain remap, but they have similar functionality through the cdn. but youd still have to have the entries set right17:59
notmynameclarkb: HP and RAX (at least used to) use the same CDN glue code, so they should behave the same way18:00
jeblairlooks like it18:00
*** nati_ueno has joined #openstack-infra18:00
openstackgerritPeter Mooshammer proposed a change to openstack/requirements: adding sphinxcontrib-programoutput to global requirements
*** safron has quit IRC18:01
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Fix paths for glossary
*** rockyg has quit IRC18:03
*** thomasm has joined #openstack-infra18:04
*** BobBall is now known as BobBallAway18:06
*** salv-orlando has quit IRC18:06
clarkbnotmyname: any idea why CDN would be done that way? seems like with the tenant + project + container you can deterministically create unique keys18:07
*** nicedice has joined #openstack-infra18:07
notmynameclarkb: actually are really cool feature of it is that the edge (Akamai in both cases here) can validate requests. this can provide _huge_ wins for protecting the origin servers (ie swift)18:08
notmynameclarkb: IOW DDOS'ing RAX or HP actually turn into trying to DDOS Akamai (good luck!)18:09
*** dcramer_ has joined #openstack-infra18:09
*** pcrews has joined #openstack-infra18:09
portantenotmyname: where is the code for that cdn stuff, do you know?18:10
notmynameclarkb: but I may be reading that code wrong (I didn't spend too much tim on it). let me see if I can ask someone18:10
clarkbnotmyname: but akamai does that for you regardless of the DNS record18:10
notmynameportante: it's the link I pasted above18:10
notmynameclarkb: no, edge requests are pattern matched to an origin. and since there are multiple endpoints, the pattern match ends up being (for RAX) *.<region>.rackcdn.com18:12
*** pblaho has joined #openstack-infra18:12
*** hashar has quit IRC18:12
*** ruhe has joined #openstack-infra18:12
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Document how to delete a pad from Etherpad Lite
notmynameclarkb: where * is any one of containers that have been created. so by adding an hmac, an invalid domain can be immediately rejected without querying the origin18:13
notmynameclarkb: ie the attack would be <uuid>.<region>.rackcdn.com18:13
clarkbI see18:13
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Ensure Conditional Builder produces valid class paths
*** DennyZhang has joined #openstack-infra18:14
mordredmorning all18:18
clarkbjeblair: are etherpad(-dev), review(-dev), wiki, and the new groups servers the only servers that need bup?18:18
clarkbjeblair: maybe paste?18:18
mordredclarkb: might as well bup paste18:20
mordredclarkb: I think the db there is pretty small anyway18:20
clarkbmordred: yeah. mysqldump should be friends with drizzle right?18:20
mordredyup. well, I'd actually use drizzledump - but it should work about the same18:20
mordredclarkb: maybe we should make paste the next server to use cloud databases18:21
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Document how to delete a pad from Etherpad Lite
clarkbmordred: wfm. To go a completely different direction is paste something we should continue to run?18:22
*** Ryan_Lane has quit IRC18:22
mordredwell, I use it18:22
mordredand it doesn't take a lot of admin resources from us18:22
*** afazekas is now known as afazekas_zz18:23
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Document how to delete a pad from Etherpad Lite
* fungi is having a bad spelling day. have to remember to 'set spell' in vim18:26
fungiguess i ought to just have it always turn that on for rst and md files18:26
*** hashar has joined #openstack-infra18:28
*** Ryan_Lane has joined #openstack-infra18:28
*** vipul is now known as vipul-away18:30
*** UtahDave has joined #openstack-infra18:31
*** melwitt has joined #openstack-infra18:31
openstackgerritA change was merged to openstack-infra/git-review: Document source location as
clarkbfungi: but then we don't get to enjoy fun tyops18:33
fungiclarkb: i promise i'll still make plenty of fun typos (more than my fair share, probably)18:34
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Fix publishing of High Availibility Guide
*** matty_dubs|lunch is now known as matty_dubs18:34
*** yolanda has quit IRC18:35
*** vipul-away is now known as vipul18:37
*** atiwari has quit IRC18:39
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Minify jquery visibility
*** atiwari has joined #openstack-infra18:40
*** mrmartin has joined #openstack-infra18:41
dkranzclarkb: Did you see the query from sbaker in this channel about why is failing?18:42
jeblairclarkb: i'm not sure we should worry too much about the dev servers18:42
clarkbdkranz: I did not18:42
*** rfolco has joined #openstack-infra18:42
clarkbjeblair: ok18:42
jeblairclarkb: i just started with jenkins-dev so i didn't step on any toes18:42
dkranzclarkb: You gave a +1 but jenkins fails with what looks like a real error18:42
dkranzclarkb: But we don't understand its nature.18:42
clarkbjeblair: fungi ^ that could use review18:43
jeblairme too18:44
*** Bada has quit IRC18:44
dkranzclarkb: I was pointing to
clarkbdkranz: yes, 46107 fixes the problem 45967 has18:44
lifelessjeblair: when convenient I have a q on
dkranzclarkb: Ah, I see. Great.18:44
clarkbdkranz: should explain the issue as well18:44
jeblairmordred, clarkb, fungi: dev-requirements will not get sync'd?18:45
fungiclarkb: on the topic of last night's bouncy castle discussion, i noticed that not only are we carting around bcprov-jdk16-144.jar but also mysql-connector-java-5.1.10.jar (seeing what i can do to get it added to puppet)18:45
*** Bada has joined #openstack-infra18:45
clarkbfungi: for bcprov I think you can do what you did for bcpg. not sure about mysql-connector18:46
jeblairlifeless: what's up?18:46
fungiclarkb: yeah, i think so but i'm checking to make sure gerrit isn't married to specific revs of those18:46
clarkbjeblair: good point, I think we may need global-requirements and dev-requirements in the list of reqs files18:46
clarkbmordred: ^18:46
fungimmm, dev-requirements is a good question... that's where we were stuffing non-pypi tarball urls from now on right?18:47
lifelessjeblair: I put it n the comments, but - I basically need guidance about what will make the patch ok; I don't have your environment or your dns tool to test with :)18:48
hasharhey there :-]  I am trying to package up the 'gear' python module, but get blocked by setup.cfg missing a version: "": Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository."""18:48
hasharhas anyone ever packaged a pbr based module using a source tar ball ?18:48
hasharthe lame error I got is :D18:49
mordredjeblair, clarkb: what fungi said - dev-requirements is just a place to put non-pypi tarball links which are under our control already18:49
*** yolanda has joined #openstack-infra18:49
mordredhashar: yes18:49
mordredyou need to use an actual source tarball though18:49
jeblairmordred: so it doesn't get synced?  and that job shouldn't test it?18:49
mordredhashar: how did you get your tarball?18:49
hasharmordred: I am using git build package which extract the tar ball in some build area :/18:50
mordredjeblair: it does get synced - I'm torn as to whether that job should test it18:50
mordredhashar: if you are using git buildpackage- you're going to need to inject a generated tarball using the import-orig command18:50
jeblairmordred: how could testing it hurt?18:50
*** KennethWilke has quit IRC18:51
mordredjeblair: fair enough18:51
fungihashar: a bunch of pbr-based openstack components are already packaged by GheRivero and zigo for debian (under the debian-openstack alioth team)18:51
clarkbmrodden1: sorry got busy18:51
fungihashar: and i think i remember they have a wiki up describing their packaging workflow, though it may not be current18:51
hasharmordred: will tried out.  I simply copied pushed your master branch to my upstream branch :D18:51
clarkbmrodden1: I think that is related to setuptools lazy loading itself onto distutils in orders that sometimes do not work18:52
reed do we really have around 100 comments per hour added on gerrit?18:52
*** yolanda has quit IRC18:52
hasharfungi: ahh I should have a look at that.18:52
clarkbmrodden1: can you try with setuptools 0.7.8 I have had good luck with that18:52
fungireed: on a slow day ;)18:52
mordredhashar: the issue is that you _either_ need the upstream git repo of you need the PKG-INFO file that's generated at sdist time18:52
fungireed: note that those include test results coming back from ci18:52
mordredmrodden1, clarkb: is there a setuptools issue?18:52
reedyeah, I imagined it's not just people :)18:53
fungireed: since they end up as review comments18:53
clarkbmordred: it has symptoms similar to the thing that clayg was working on18:53
reedaround 20 patches per hour is an interesting number too18:53
mrodden1clarkb: trying with 0.7.818:54
jog0anyone have a minute to review my 3rd attempt to unbreak rechekwatch
mrodden1same output... :(18:54
jog0(that I broke ... sorry)18:54
clarkbmrodden1: hmm maybe not related then18:55
clarkbmordred: also unrelated but interesting bug 1206730. I believe your merge milestone proposed branches back into master is the fix18:55
uvirtbotLaunchpad bug 1206730 in pbr "pbr generates versions that are not monotonic" [Undecided,New]
mordredmrodden1: is this just on a plain devstack?18:56
mordredclarkb: it's not18:56
*** lnxnut_ has joined #openstack-infra18:56
mrodden1well its kind of an old install18:56
mordredthe versions will still not be monotonically increasing18:56
*** thomasbiege has quit IRC18:56
mordredmrodden1: shouldn't matter - it shoudl work18:56
mrodden1i updated nova today and tried to do develop and it not happy18:56
clarkbmordred: don't we count after last tag?18:56
mordredclarkb: no18:56
mordrednot for server projects18:56
clarkbmordred: so you get $base_version.incrementing_number.sha_thing18:56
*** bswrchrd_ has joined #openstack-infra18:56
clarkbmordred: oh18:56
mordredwe use pre-version numbering18:56
jeblairlifeless: i don't have a good answer for you; however, i would like it to generate output that works for us, and not generate output that does not work for us.  the current output does work for us.18:57
clarkbpre-version doesn't prevent you from doing that I thought18:57
openstackgerritA change was merged to openstack-infra/config: Update old bug format in recheckwatch
jog0thanks mordred18:57
clarkboh except those versions will be greater than the base18:57
mordredclarkb: the versions have no relation to previous tags18:57
lifelessjeblair: ok. I can make putting the domain on the rcreate calls optional18:57
jeblairlifeless: that should work18:58
mordredso you'll have 2013.1.a1.g234523 then 2013.1.b1 (produced by tag) then 2013.1.a3.g23422518:58
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Puppet the Gerrit bcprov and mysql-connector deps
mrodden1brb need to step away for a bit...18:58
mordredmrodden1: ok. I'd love to figure out why it broke - I'm fairly confident I can get you fixed again18:58
*** bswrchrd has quit IRC18:59
*** bswrchrd has joined #openstack-infra19:03
*** changbl has joined #openstack-infra19:05
*** bswrchrd_ has quit IRC19:05
*** ruhe has quit IRC19:08
*** fbo_away is now known as fbo19:12
clarkbfungi: re and the change that "fixes" ssh hash mismatch errors. I can squash them together. The one that deals with bouncy castle needs work though19:14
clarkbfungi: I am beginning to think that one central gerrit server should be used and we create a project per test19:14
fungiyeah, we could potentially just reorder those patches too19:15
clarkbfungi: because gerrit start up time is insane with bouncy castle19:15
clarkbfungi: I have seen that 5 minute timeout not be long enough on my test box19:15
fungii take it bc wasn't actually getting used i the current rev of that task then?19:15
mordredclarkb: might be a good opportunity to look at testresources for that then19:15
clarkbfungi: correct, because the lib isn't installed globally on the slaves19:15
fungibecause the tests actually completed in about the same time19:15
clarkbmordred: yup19:15
fungii overlooked the bit where it wasn't in place in that patch. no wonder19:16
clarkbfungi: so basically prep the golden site, spin it up and tear it down with a testresource19:16
clarkband do away with per test gerrit servers19:16
fungithat makes more sense, yes19:16
fungiso only one gerrit install happens per job run that way19:17
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Fix paths for glossary
clarkbfungi: right19:17
clarkbfungi: the slowness appears to be at least partially load related19:17
clarkbfungi: spinning up one gerrit is typically ok. spinning up 4 at the same time is not (I assume IO related?)19:17
fungidoes not surprise me too much19:18
*** salv-orlando has joined #openstack-infra19:18
*** vipul is now known as vipul-away19:23
*** vipul-away is now known as vipul19:23
mordredjeblair: on
mordredjeblair: dev-requirements is already tested in that file19:28
mordredjeblair: it has a block later in the script19:28
*** gyee has quit IRC19:29
mrodden1mordred: back, sorry had to run out to fix something else quick19:30
mrodden1i'm wondering if its a python path issue but i'm not positive19:31
*** alexpilotti has quit IRC19:32
mordredmrodden1: so, newer devstack has some workarounds for a setuptools upgrade issue19:35
mordredmrodden1: try this "pip uninstall setuptools; pip install -U setuptools; pip install -U pip; pip install -U pbr"19:35
mordredyea - there was some carnage caused by upstream re-merging of setuptools and distribute19:36
annegentlehey infra, did anyone coach some of the training guys (sarob or colin) to do a subproject within openstack-manuals repo?19:36
*** Ryan_Lane has quit IRC19:36
annegentleI'm really clueless and trying to figure out what's up with some git history19:37
jeblairmordred: commit message bug then?19:37
jeblairannegentle: i haven't caught on to what you're asking yet... is there a git commit i should look at?19:37
mordredannegentle: we never suggest subprojects to people - but also, I agree with jeblair in that I don't fully follow you19:38
jeblairmordred: annegentle may not have meant 'git submodule' which i assume is what you were thinking19:38
annegentlejeblair: just now spelunking as a new src folder has appeared19:38
funginot to my knowledge anyway19:38
annegentlejeblair: c748436dd23da10861a98da63440acba23b39e01 is one19:38
*** vipul is now known as vipul-away19:38
annegentlejeblair: 090a2dfb1ca701ef31cf30579628c083002894ed is another, those are commit numbers, I can also give you Change-IDs19:38
annegentlejeblair: yeah I don't mean submodule19:39
jeblairannegentle: i brainstormed some ideas with sarob19:39
annegentlejeblair: ok19:40
annegentlejeblair: has these extra directories19:40
jeblairat least, i assume i did; i brainstormed with sean roberts, whose name looks like sarob19:40
jeblairooh, self-approved organizational changes!19:40
mordredtime for some public shame perhpas19:41
annegentlejeblair: yes sarob is sean roberts19:41
jeblairannegentle: i _definitely_ did not suggest anything that involved "go to trello"19:41
annegentlejeblair: well they are "incubating" under docs... and no I haven't asked them to stop using trello :)19:42
annegentlejeblair: this is all highly ungoverned :)19:42
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Publish Over SSH Plugin option added
mordred,unified <-- what is that? what's a Subproject?19:42
annegentlemordred: I dunno either19:42
*** Ajaeger has joined #openstack-infra19:42
annegentlemordred: their goal is: use the existing docs to make custom training manuals19:43
annegentlemordred: and I LIKE that goal19:43
annegentlemordred: it'll make our docs better too19:43
jeblairannegentle: i suggested some ideas around how they could automatically pull in stuff from other repos by using some technology similar to what mordred is working on for auto-proposing requirements changes19:43
annegentlemordred: but, achieving "give me these bits and pieces" takes some work19:43
jeblairannegentle: (where a change to a file in one repo would trigger a job that copies the change to another repo)19:43
mordredah. interesting19:44
annegentlejeblair: oo! mordred! I need that for "watching" the nova api samples directory!19:44
fungisubprojects do in fact seem to bear some relation to submodules19:44
jeblair(copying files around to different repos is, of course, insane, but i'm not sure how you release a documentation snippet as a library, so hey)19:44
*** adalbas has quit IRC19:44
jeblair(and insane has never stopped us from trying an idea)19:44
annegentlejeblair: hee19:45
annegentleless sanity, more experimentation19:45
*** zul has quit IRC19:45
jeblairfungi: ok, so a subproject commit is actually a git submodule19:45
jeblairthat's great that they have two names19:45
mordredthat's stellar19:45
fungijeblair: beware the hobgoblin19:46
mordredso, that means we're using submodules now in the doc repo?19:46
annegentlemordred: the use case is: nova dev makes some updates to api samples, the openstack/api-site automatically gets those new samples and makes a new patch and builds new docs with fresher samples19:46
jeblairmordred: well, someone is; i doubt "we" are.19:46
*** adalbas has joined #openstack-infra19:46
annegentlemordred: do not want. submodules.19:46
mordredsorry, I mean "we" are using submodules in the doc repo now?19:46
annegentlemordred: do. not. want :)19:46
jeblairwe don't want it either.  no one knows how to use submodules.  even people who think they do.19:46
annegentlejeblair: zactly19:46
mordredyup. also, zuul does not know how to use submodules19:47
*** hashar has quit IRC19:47
*** dcramer_ has quit IRC19:48
fungii think one of the suggestions i had was that the api samples in the software projects should have associated publish jobs to upload them separately from the docs jobs, or that the docs jobs should depend on checkouts of the software projects containing the api samples. but maybe automated embedding of that stuff isn't straightforward19:49
fungibut at the time it was all still very brainstormy19:49
mordredthey're only half there19:49
annegentleand I don't mean to conflate the two goals/use cases -- API samples are different from custom training manuals...19:49
jeblairfungi: yeah, any of those things sounds like a good direction to explore19:49
mordredthe .gitmodules file is missing19:49
annegentlemordred: that's good right?19:50
annegentlemordred: as in, reversible?19:50
annegentleI wish sarob or colin were here to talk to... I'll email them I think.19:50
fungiannegentle: yeah, clearly the training manuals case is different from the api docs case, but it seemed to share some similar needs19:51
AjaegerAnd they committed to the wrong directory, doc/src/docbkx is obsolete.19:51
annegentlefungi: for sure.19:51
jeblairmordred: given that we've established no one understand submodules, that means there's no one that could add support for them to zuul.  :)19:51
* fungi suspects zuul will become sentient at some stage, and might add them itself then19:52
jeblairfungi: it would be the only thing that understood them then.19:52 will start getting commits directly from zuul to correct git's current submodule implementation19:53
mordredjeblair: well.... I'm going to need to add support for them to zuul at some point, but I'm deferring thinking about it for right now19:53
Ajaegerwhile you all talk manuals, could I bother you with some reviews for fallout of our big directory rename? These three are needed for getting everything published:
clarkbI just found out that you couldn't cleanly remove submodules until git 1.8.319:54
*** SergeyLukjanov has quit IRC19:54
jeblairmordred: good luck!19:55
mordredjeblair: do you want me to rework that commit message? I'd like to land the requirements fix19:55
*** DennyZhang has quit IRC19:55
mordredjeblair: thanks!19:55
annegentleThanks Ajaeger for the patch to remove, I'll see if I can get Sean to take a look19:55
jeblairmordred: i kinda would like you to, it is pretty wrong...19:56
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove old files from the requirements list
jeblairmordred: i'll review it all quick-like19:56
mordredhow about that?19:56
*** zul has joined #openstack-infra19:59
openstackgerritA change was merged to openstack-infra/config: Remove old files from the requirements list
*** mriedem has quit IRC19:59
*** mriedem1 has joined #openstack-infra20:00
*** bswrchrd has quit IRC20:01
*** mriedem1 has quit IRC20:01
*** pcm__ has quit IRC20:01
*** mriedem has joined #openstack-infra20:03
*** jcoufal has quit IRC20:03
*** HenryG_ has quit IRC20:03
fungimordred: looking at 40068 with an eye for extending it to stable branches, would it make more sense to relocate that list of projects to which requirements changes should be autoproposed out of the slave script and into a file in openstack/requirements instead? (since it will change over time between releases)20:03
mordredhrm. that's not a bad idea20:04
fungiotherwise we start implementing devstack-gate-like branch conditionals to build up the list, which gets hairy20:04
Ajaegerjeblair, another question regarding templates for manual build jobs (see
mordredI do think it means that we may wind up with information in that repo which is not really consumed by anyone but infra - but I think I can deal with that20:04
fungidoable, but would be simpler to just key off something in the requirements repo which can differ by branch20:05
Ajaegerwill the following start two build jobs:20:05
Ajaeger- project:20:05
Ajaeger    name: openstack-manuals20:05
Ajaeger    manual:20:05
Ajaeger      - user-guide20:05
Ajaeger      - image-guide20:05
Ajaeger    jobs:20:05
Ajaeger      - '{manual}-documentation'20:05
annegentlereed: around?20:05
mrodden1mordred: i got it20:06
mrodden1i had some setuptools egg info hanging around from the .deb packages20:06
reedannegentle, wazzup?20:06
mrodden1deleted that and the pip installed version kicked in and now everything is happy20:06
annegentlereed: have a q for ya20:07
*** afazekas_zz has quit IRC20:07
annegentleAjaeger: I think so- jeblair?20:09
annegentleAjaeger: I was even at the tutorial Monday and followed it :)20:09
annegentleAjaeger: but will still ask jeblair20:09
clarkbAjaeger: annegentle: that looks correct to me20:09
openstackgerritMonty Taylor proposed a change to openstack/requirements: Add a list of projects that we want to sync to
mordredfungi: ^^20:09
clarkbas JJB should iterate the manual values automagically20:09
Ajaegerannegentle: was the tutorial recorded or is material available from it?20:10
fungimordred: awesome!20:10
Ajaegerclarkb, annegentle thansk for the confirmation! That brings me a step further to simplifiy the jjb manual jobs20:10
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Update projects when requirements change
mordredfungi: updated to consume it20:10
annegentleAjaeger: it'll be out in a week or two20:11
Ajaegerannegentle: great, thanks20:11
*** weshay has quit IRC20:12
*** jcoufal has joined #openstack-infra20:12
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Update projects when requirements change
fungimordred: also, just from a separation of procedure and data perspective, i like that we don't have yet-another-list-embedded-in-a-script syndrome there20:13
mordredfungi: ++20:13
mordredgreat idea20:13
jeblairAjaeger: there's the commands from the lab; if you run those, you'll end up with a local copy of jenkins-job builder, and you can use it to test out the syntax20:14
jeblairAjaeger: (if you use it as described, you'll end up with a directory with one xml file for each job, so you can see if it's creating the jobs you expect)20:14
*** mgagne1 has joined #openstack-infra20:15
*** mgagne1 has joined #openstack-infra20:15
*** mgagne has quit IRC20:16
openstackgerritMonty Taylor proposed a change to openstack-infra/pypi-mirror: Remove spurious uses of output return values
*** fbo is now known as fbo_away20:16
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Start using templates in manual build jobs
Ajaegerjeblair: Thanks, will test later.20:19
*** weshay has joined #openstack-infra20:21
*** niska has quit IRC20:24
mordredjeblair, clarkb, fungi: if you aren't subscribed to openstack-infra/pypi-mirror, now might be a good time :)20:24
fungioof--good reminder20:25
fungii should re-audit my subscriptions against cgit's openstack-infra list20:25
jeblairAjaeger: /win 2320:25
*** niska has joined #openstack-infra20:26
fungifull of win20:26
*** dprince has quit IRC20:28
*** dcramer_ has joined #openstack-infra20:29
clarkbI think claws wins the current MUA competition20:30
clarkbalso I may have spent lunch derping over MUAs20:30
*** lnxnut_ has quit IRC20:30
fungii have not tried claws. you liked it better than gnus huh?20:31
clarkbI am not super comfortable with emacs20:32
fungithat would definitely make gnus less attractive20:32
clarkbI can't get it to verify ttx's signature though20:33
clarkbbut I think that may be because the uids don't match?20:33
fungithat shouldn't matter20:33
clarkbhmm, needs more hammering then20:33
fungias long as the public part of the signing key's parent is in your keychain, it should at least tell you the signature is valid, even if not valid for the sender's address20:34
fungiwas the message signed inline or multi-part pgp+mime?20:35
clarkboh I see the error now. His key is not available to verify the signature20:35
clarkbprobably keychain issues20:35
clarkbfungi: inline I believe (one of the recent CVE announcements)20:35
fungiyeah, his client seems to sign inline unless there are attachments20:36
fungiwhereas mutt always uses pgp+mime by default for me20:36
fungii also don't have mutt configured to validate inline signatures, but it's only a |gpgv away20:36
Ajaegerjeblair: thanks for the step-for-step instructions, now I can test my changes!20:39
jeblairyay!  i should put those in the jjb docs20:39
*** KennethWilke has joined #openstack-infra20:40
*** pblaho has quit IRC20:42
mordredI just got the best unsolicited recruiter email ever20:42
mordredNeed Java Architect  – Google Web Technologies (GWT) expert20:42
mordredyeah. THAT's me20:42
*** vipul-away is now known as vipul20:42
*** Ryan_Lane has joined #openstack-infra20:43
*** Ryan_Lane has joined #openstack-infra20:43
ekarlso-jeblair: what's a way to have zuul notify a vendor jenkins upon change ?20:45
Shrewsmordred: OMG. PLZ TO SEND TO ME20:45
clarkbShrews: mordred hahaha20:45
jeblairekarlso-: there isn't; see this instead:
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Start using templates in manual build jobs
ekarlso-jeblair: would be cool if there was a feed or so from zuul20:49
jeblairekarlso-: there is a feed from gerrit20:50
ekarlso-ok :)20:50
Ajaegerjeblair: thanks to the testing, 45574 is valid now ;) I would appreciate some review comments on it before I continue.20:50
jeblairAjaeger: yay!20:52
*** mrmartin has quit IRC20:56
*** nhm has joined #openstack-infra20:59
*** mriedem1 has joined #openstack-infra21:00
Ajaegerjeblair: thanks, will do another iteration tomorrow. Bye for now!21:00
*** hashar has joined #openstack-infra21:00
*** Ajaeger has quit IRC21:01
*** mriedem has quit IRC21:02
*** ArxCruz has quit IRC21:02
*** zeus has quit IRC21:03
*** rfolco has quit IRC21:04
*** Bada has quit IRC21:07
*** gyee has joined #openstack-infra21:08
openstackgerritClark Boylan proposed a change to openstack-infra/publications: Make make-index ignore template branch.
*** dmakogon_ has joined #openstack-infra21:09
clarkbfungi: jeblair mordred ^ that is a start to creating a template branch for publications and writing a howto create new publication21:10
clarkbI haven't created the template branch or anything else yet :)21:11
clarkbbut working on it21:11
*** mgagne1 has quit IRC21:11
*** dkliban has quit IRC21:12
jog0so I am still failing at getting unbreaking rechecks :(21:14
hasharfungi: mordred: I found out how to get the python module version injected in Debian packaging tool :-)  Got to include the openstack packaging tool then do something like: OSLO_PACKAGE_VERSION=$(VERSION)  :-]21:18
fungihashar: yes, that's one of several backdoor solutions mordred added for packagers who want to override versioning21:18
hasharfungi: that went well for me, I am building from a the source tar :-]21:19
hasharnow I get to backport all your dependencies huhu (d2to1 .. pbr …)21:19
fungihashar: i thought those were already packaged for debian21:20
*** svarnau has quit IRC21:20
hasharyup, got to get them backported to Ubuntu Precise which is what is being used by Wikimedia21:20
*** dcramer_ has quit IRC21:20
fungithey may still be in the new queue or experimental/sid/testing...21:20
jog0clarkb: have a minute to put me back on the right path for not breaking things?21:20
fungihashar: ahh, right. shouldn't be too tough then21:21
hasharfungi: they landed in saucy already iirc. I will be fine (hopefully)21:21
lifelessdid just die ?21:21
hashar!status gerrit21:21
openstackhashar: Error: "status" is not a valid command.21:21
jeblairlifeless: no21:21
SpamapSI think is having issues .. from my local box here in LA ..   Could not fetch URL timed out21:21
fungiseems up to me21:21
lifeless19:                            306.860ms asymm 1821:21
lifeless20:                                511.968ms !H21:21
hasharlifeless: gerrit web interface works for me21:21
lifeless     Resume: pmtu 150021:21
*** pycabrera has left #openstack-infra21:21
*** dkranz has quit IRC21:21
fungiooh, that's fun21:21
*** lcestari has quit IRC21:22
lifelessfungi: FSVO21:22
hasharlifeless: ssh works for me as well21:22
fungii'm getting to it via core3 as well and getting an actual reply21:22
fungialso via ipv6 through whatever's at 2001:4800:800:c3:150a:4:0:1 as the penultimate hop21:23
lifelessfungi: sadly my ISP is not enlightened21:23
lifelessso I can't check ipv621:23
*** weshay has quit IRC21:24
lifelessI'm also getting21:24
lifeless20:                            307.901ms asymm 1821:24
jeblairi am seeing some ipv6 packet loss on rax's network21:24
lifeless21:                                716.756ms !H21:24
fungilifeless: neither is mine. i tunnel to hurricane electric21:24
lifelessso this seems larger than that one host.21:24
lifeless     Resume: pmtu 150021:24
fungithey're at least not admitting to any issues, or haven't gotten around to doing so at any rate21:25
lifelessfungi: can you see if you see traffic from me ?21:26
*** senk has joined #openstack-infra21:26
fungilifeless: your source ip address is...?21:26
lifelessit just came good21:27
fungioh well21:27
fungii doubt it was an issue along an asymmetric return path unless rackspace is making internet backbone choices several hops into their core21:28
*** senk has quit IRC21:28
fungicould have been flow hash related though, and only exhibited for certain remote addresses21:28
fungior some provider closer to you was blackholing some of rackspace's ip space which included some of our servers but not those core routers (they are on significantly different parts of the ipv4 global address space)21:30
fungithis is the point where, if i were still a network engineer at an isp, i would be digging through looking for whacky peering transitions out on the 'net and posting annoyed messages to the nanog ml21:32
fungiso glad i'm not ;)21:32
lifelessthat seems possible, since just when gnaaargh21:32
jog0fungi: ping21:36
*** mriedem1 has quit IRC21:36
fungijog0: howdy21:36
fungijog0: still troubleshooting rechecks?21:36
jog0fungi: sigh ... yeah21:36
fungijog0: any leads?21:37
jog0I was wondering if you could confirm that it tried to restart today21:37
fungitaking a look now21:37
jog0I tested the latest patch locally and it appeared to work21:37
jog0with the pickle jeblair gave me21:37
jog0fungi: thanks for your patience21:37
*** KennethWilke has quit IRC21:38
fungiSep 12 19:00:44 zuul puppet-agent[12549]: (/Stage[main]/Recheckwatch/File[/usr/local/bin/recheckwatch]/content) content changed '{md5}f649e84163517a28e0821eee382c6cdf' to '{md5}ceccb0659de7cf6e3c153eab79d8df4c'21:39
fungiSep 12 19:00:48 zuul puppet-agent[12549]: (/Stage[main]/Recheckwatch/Service[recheckwatch]) Triggered 'refresh' from 1 events21:39
fungithough it may not have been running, so the service refresh may have been a no-op (can't remember if puppet would complain in that situation)21:39
jog0I assume the process isn't running now though?21:39
clarkbfungi: jeblair: is 'template' a good name for the publications template branch? if so I am going to create that branch and push a change to it21:40
fungijust started it back up, jog021:40
jog0can you check in a minute or two if its still running21:40
jeblairclarkb: wfm21:40
fungijog0: seems to have stayed running for the moment, but yeah i'll take another look in a bit21:40
jog0fungi: thanks if it fails it will take a few minutes21:41
fungiclarkb: yes, i'm 50/50 on template vs skeleton, but the former probably translates better outside of english21:41
fungiarchetype would also work21:41
fungibut is a little on the unnecessarily fancy side21:41
clarkbtemplate it is then21:41
fungisounds good21:41
*** markmcclain1 has quit IRC21:42
*** hashar has quit IRC21:45
clarkbfungi: jeblair mordred pleia2 zaro
mrodden1so.. now when i on my precise machine... brings in old versions of ubuntu packages that are incompatible with nova requirements21:47
clarkbI based template on overview so that I could hack 'n slash the existing content. This also gives you good examples in git history21:47
mrodden1and that overrides the pip installed versions in /usr/local/21:47
clarkbmrodden1: /usr/local/ should win with default python path21:47
jog0fungi: it appears to be working, can you make sure the process is still up as well21:49
fungijog0: just did. seems to have survived the first 10 minutes at least and is still chugging21:49
mrodden1clarkb: that doesn't appear to be the case on my system21:49
*** danger_fo is now known as danger_fo_away21:49
mrodden1probably need to figure out how that got messed up i guess21:50
jog0fungi: thanks!21:50
clarkbmrodden1: is this ubuntu or debian?21:50
zaroclarkb: nice work.  but i probably will never branch from template.21:50
clarkbzaro: why not?21:50
*** hashar has joined #openstack-infra21:51
zaroclarkb: prefer to work from more to less.  than other way around.21:51
clarkbok, just making sure that I wasn't missing something significant21:51
fungiin which case you can s/template/overview/ in those instructions and it'll still be mostly relevant21:51
clarkbfungi: zaro: also I branched template from overview so you can just look in history (this was intentional)21:52
fungiyep, sounded like a good move21:52
zaroyes, nice to have options.21:52
fungii had already done most of the cleanup getting cruft files blown away in the first iterations of the overview branch creation, so should be pretty similar21:53
*** thomasm has quit IRC21:53
mrodden1wth... i had /usr/lib/python2.7/dist-packages in my easy-install.pth in /usr/local/lib/python2.721:55
fungithat sounds marginally recursive21:55
mrodden1now that that is fixed... lets try this again (
*** boris-42 has quit IRC21:57
*** changbl has quit IRC22:00
lifelessfungi: ok its broke again22:04
lifelesswhee now I don't even get dns22:04
lifelessfungi: ah, it was broken due to dns this time. verra odd22:05
jog0fungi: recheck appears to still be working thanks again22:06
openstackgerritKhai Do proposed a change to openstack-infra/config: Diff between installed packages and requirements
jog0fungi:now to get my second patch for recheckick in without breaking things again22:12
*** lifeless_ has joined #openstack-infra22:13
*** lifeless has quit IRC22:15
*** lifeless_ is now known as lifeless22:18
*** jcoufal has quit IRC22:24
fungilifeless: i'm surprised to hear that the internet is not a bastion of stability and reliability ;)22:25
fungijog0: great!22:26
clarkbfungi: re would bash string manipulation be safer than the backticks and sed?22:27
clarkboh I see it isn't the sed that is a problem22:27
*** flaper87 is now known as flaper87|afk22:28
clarkbits that the resulting of sedding might be mytag ; rm -rf22:28
fungiclarkb: yeah, or whatever22:28
fungibut i really couldn't get it to not sanely escape all the things i fed into a tag22:28
fungialso, what can go into a tag is a little limited... man git-check-ref-format22:29
*** atiwari has quit IRC22:29
pleia2clarkb: thanks for writing these instructions22:29
fungiin particular, spaces aren't allowed which mostly limits you to running a single name (but could still be a file in your repo)22:29
*** markmcclain has joined #openstack-infra22:30
clarkbpleia2: np, let me know if they need more depth or if I got something wrong22:30
pleia2clarkb: once this is merged I'll give the instructions a try with mine and let you know ;)22:30
pleia2seems good though22:31
clarkbzaro: was rebased onto mordred fix?22:31
clarkbzaro: looks like it was22:32
lifelessfungi: droll, very droll :)22:33
lifelesswho do I talk to about making sure requirements files do /not/ list URLs ?22:34
*** hashar has quit IRC22:34
*** dmakogon_ has quit IRC22:34
clarkblifeless: mordred and sdage22:34
lifelessmordred: sdague: ^ oh HAI22:34
*** matty_dubs is now known as matty_dubs|gone22:35
zaroclarkb: yes, but i kept the orig implmentation of because it can handle multiple requirements files.22:36
clarkbzaro: ok22:36
*** jhesketh has joined #openstack-infra22:37
*** thedodd has quit IRC22:38
*** mrodden1 has quit IRC22:39
*** pentameter has quit IRC22:39
openstackgerritA change was merged to openstack-infra/jeepyb: Update doc impact to only make bugs on merge
*** jhesketh__ has joined #openstack-infra22:42
*** zoresvit has joined #openstack-infra22:43
zoresvitHi. Can anyone tell if Repo Hooks API is available in GitHub Enterprise? The Enterprise version is not the latest...22:44
*** KLevenstein has quit IRC22:45
fungizoresvit: i think you may be looking for github's support people. pretty sure none of them hang out in here, though i could be wrong22:46
clarkbzoresvit: I wouldn't expect anyone here to know that. I can point you to the script that clsoes our pull requests
zoresvitOk, got it! Thanks a lot for this as well.22:47
clarkbre I think we need to limit the test matrix at some point...22:50
*** pcrews has quit IRC22:53
*** zoresvit has quit IRC22:55
*** zoresvit has joined #openstack-infra22:56
*** mrodden has joined #openstack-infra23:00
fungii think we raised that concern two summits ago when people wanted to start testing with postgres?23:00
*** locke105 has joined #openstack-infra23:03
*** UtahDave has quit IRC23:05
*** hashar has joined #openstack-infra23:09
anteayaso we only have one more day of UtahDave's dedicated time. I want to make sure we don't get distracted with other stuff and lose him23:11
*** adalbas has quit IRC23:11
anteayafungi: what is the current salt status?23:12
anteayaI've been trying to keep track but I might have missed something23:12
anteayayou are currently ensuring that we can get the latest salt on all our slaves, is that correct?23:12
fungianteaya: i have a stack of salt-relevant reviews up, and i'm in the process of editing his to add it on top of them23:13
anteayajust what I was looking for, thanks23:14
*** rnirmal has quit IRC23:18
*** mrodden has quit IRC23:20
*** mrodden has joined #openstack-infra23:20
*** mrodden has quit IRC23:21
*** mgagne has joined #openstack-infra23:24
*** mgagne has joined #openstack-infra23:24
*** mgagne1 has joined #openstack-infra23:33
*** mgagne1 has joined #openstack-infra23:33
clarkbanteaya: I will look at those changes as well23:35
*** mgagne has quit IRC23:36
clarkbfungi: we did raise the concern. I don't think we have hit an upper limit either, I am just reminded that this is a thing whenever changes like jd__'s come in23:36
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Give Jenkins the ability to trigger puppet runs
fungianteaya: clarkb: added ^ (the rebase/adjustment of UtahDave's patch) just now23:38
fungineed to follow that one with a jenkins job definition and script to run it23:38
clarkbfungi: re wouldn't it be better to configure auto updates for salt in the salt module?23:38
*** mrodden has joined #openstack-infra23:38
clarkbeg instaed of in slave.pp?23:38
clarkbor is that an issue because we can configure unattended upgrades in one place? no conf.d/*?23:39
fungiclarkb: oh, perhaps... can modules set things in each other in a transverse fashion like that?23:39
fungiclarkb: and yeah, it needs to be included into the block in that one file23:39
clarkbfungi: it isn;t easy when it must go in the one file23:40
fungican't be in a separate file i don't think23:40
clarkbyou have to use puppet concat and things get weird23:40
clarkbfungi: isn't 50unattendedupgrades in a conf.d?23:40
clarkbcan we go another level deep and have that file include the others? or is that particular file magical?23:41
*** hashar has quit IRC23:41
fungioh, good point, we could probably break that file up into a 50* and a 60* and split them where we want to insert lines23:41
fungiand then have the salt module add a 55* which includes just the one line we want23:42
* fungi looks23:42
clarkbfungi: if that isn't too difficult I think it would be a good way to go23:42
clarkbfungi: the salt module can then include unattended_upgrades and require that class before creating the files23:43
clarkband that should keep everything in the correct order23:43
fungiwell, yes except we still need unattended upgrades on systems where we don't use salt, right? so we'd need to apply it differently to the other systems to avoid include conflicts in puppet23:44
clarkbfungi: there won't be conflicts if you use include23:44
clarkbyou can include a thing as many times as you want and the first one wins.23:44
fungiahh, okay. it's just class instantiation which goes sideways if you've already included the same?23:45
clarkbinclude is special23:46
fungiso as long as we only include and don't need to pass in variables we should be safe23:47
fungii'll finally comprehend puppet about the time we switch to something else23:48
*** dims has quit IRC23:52
*** Ryan_Lane has quit IRC23:56
*** hashar has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!