Wednesday, 2013-10-09

fungithought might as well be start the review00:00
*** oubiwann_ has quit IRC00:00
lifelessclarkb: bash -xe $BASE/new/pbr/tools/ $PROJECTS $BASE00:00
lifelessclarkb: from ./modules/openstack_project/files/jenkins_job_builder/config/pbr.yaml00:00
lifelessclarkb: which still uses the mirror according to script - at least how I read it00:00
clarkbfungi: why did the cells jobs not get {release} templated too? I get why the folsom jobs are not templates00:01
*** mrodden has joined #openstack-infra00:01
fungii templated any jobs which are run for grizzly and also master (thus presumably soon havana)00:02
clarkbmordred: does the raw install job use the mirror?00:02
*** DennyZhang has quit IRC00:02
clarkbfungi: the cells jobs should be run for both00:02
fungimmm, i might have overlooked that (there's more than one cells job too though)00:02
mordredclarkb: yes00:02
mordredit uses it in extra_index_url00:03
mordredotherwise it does't work00:03
mordredbecause our mirror actually has historical things that are there because they _were_ requirements but are no longer expressed in our file00:03
clarkbfungi: right there are three (we really only need one and incrementally switch to the additional jobs as things start to work00:03
mordredbecause of stepwise migrations00:03
clarkbfungi: I am double checking cells now. it definitely runs against master00:03
mordredbut, xattr and pycparsing happily install from our mirror00:03
clarkbfungi: looks like it excludes folsom so grizzly and master00:04
lifelessso, I have three pbr patches00:04
lifelessI would like to get off my plate00:04
mordredI agree00:04
mordredclark has rechecked one of them00:04
lifelesscan I ask a regular pbr contributor to drive them through perhaps?00:04
dstufftissues with the CDN?00:04
lifelessmordred: so have I, repeatedly.00:04
clarkbdstufft: maybe not, I think I was mistaken00:05
*** julim has quit IRC00:05
lifelessmordred: I have no confidence this will fix it00:05
dstufftclarkb: ok :D I prefer mistaken to broken :]00:05
lifelessmordred: I will happily fix any bugs in my patch itself00:05
clarkbmordred: but isn't raw install used to gate requirements now because it doesn't use the mirror?00:05
mordredhrm. so, my patch and fungi's patch to pbr both passed00:05
lifelessmordred: I just don't want to be doing rework because something else merges ahead when the issue was $other00:05
mordredclarkb: it builds its own mirror00:06
clarkbmordred: from pypi.python or pypi.o.o?00:06
mordredwell - funy story00:06
*** mdenny has quit IRC00:06
mordredit USED to build its local mirror from our mirror00:06
mordredbut jeremy and I both have recently modified it to remove that chunk of code from integration.sh00:06
mordredneither patch has landed yet00:07
clarkbso lifeless' failed tests would not have touched pypi.python?00:07
mordredand they don't - that's not what's causing his patch to fail00:07
mordredwhat's causing his patch to fail is that, for reasons making my brain want to bleed, the local mirror in the test does not have a copy of pycparsin00:07
fungiclarkb: ahh, yeah all three of check-devstack-vm-cells, check-tempest-devstack-vm-cells and check-tempest-devstack-vm-cells-full are !stable/folsom so should be templated. i'll rework that00:07
clarkbfungi: thank you00:08
mordredI believe this is to be expected, given that pycparsing comes in via cffi setup_requires00:08
mordredI could have SWORN we used to have an entry in reuqirements listing pycparsing explicitly because we can't catch transitive setup_requires depends00:08
*** mdenny has joined #openstack-infra00:09
mordredclarkb: oh - also, no00:09
mordredclarkb: we only run pbr integration on pbr patches still00:09
mordredwe have not made it part of the requirements gate - we're still working on that00:09
mordredwe WANT to do that00:09
clarkbmordred: git log -p says no pycparsing00:10
clarkb(in requirements)00:10
*** reed has quit IRC00:10
mordredI wonder if something changed with cffi upstream. again00:10
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1235486
uvirtbotLaunchpad bug 1235486 in neutron "Integrity violation on delete network" [Medium,In progress]
*** shardy has quit IRC00:11
mordredclarkb: I'm in favor of landing fungi's
mordredclarkb: at the very least, I believe it's more correct00:13
clarkbmordred: is d-g setting that stuff up for
clarkb(is that why it is more correct? or is it more correct to not use the mirror at all?)00:15
mordredit's more correct to let d-g do the work00:15
mordredand yes00:15
fungiclarkb: be aware there's a config capstone change i've -2'd for the d-g and pbr changes00:21
clarkbfungi: link?00:21
fungiit's waiting on those to merge00:21
fungii'll dig it up--just a sec00:22
clarkbI think I found it00:22
clarkbmy initial scan didn't have results but as I started actually looking appeared00:22
fungiyou probably have, as it's hard to mistake00:22
openstackgerritA change was merged to openstack-infra/elastic-recheck: Remove query for closed bug 1223469
uvirtbotLaunchpad bug 1223469 in cinder "Volume create tgtadmin fails with error code 22" [High,Fix released]
*** jhesketh__ has quit IRC00:24
*** dims has quit IRC00:24
*** Ryan_Lane has quit IRC00:28
*** Ryan_Lane has joined #openstack-infra00:29
*** matsuhashi has joined #openstack-infra00:30
*** jhesketh__ has joined #openstack-infra00:31
clarkbhmm we need to restart gerrit to pick up the new gerrit bug regexes00:31
clarkbfungi: ^ I didn't think of that when I approved00:31
fungiprobably so, yes00:32
*** Ryan_Lane has quit IRC00:33
*** Ryan_Lane has joined #openstack-infra00:33
clarkbjog0: just got -1'd I noticed something I missed the first time around :/00:37
openstackgerritA change was merged to openstack/requirements: bumping django_openstack_auth version
jog0clarkb: Iec9162824752e11cfc4845f8c168c5c62100d18500:39
jog0that was intentional00:40
*** Ryan_Lane has quit IRC00:42
*** Ryan_Lane has joined #openstack-infra00:42
notmynamemordred: can you offer your public opinion on please?00:42
mordrednotmyname: absolutely00:42
notmynamemordred: thanks. just want an "on record" ;-)00:43
clarkbjog0: aha!00:43
clarkbjog0: I will update my vote00:43
*** Ryan_Lane has quit IRC00:44
clarkb++ to wanting a new pip anyway00:44
*** Ryan_Lane has joined #openstack-infra00:45
jog0clarkb: its an issue we need to fix at some point00:45
jog0the test *should* work on rax too00:45
lifelessmordred: transitive deps are hard mmmkay00:45
mordredlifeless: o m g00:45
clarkbjog0: understood00:45
*** changbl has joined #openstack-infra00:51
* clarkb -> home00:52
mordredlifeless: they aren't quite as hard - but transitive setup_requires are the worst00:54
mordredbecause you can't query them without running the setup.py00:54
jeblairmordred: i made pootle support openid:
*** erfanian has joined #openstack-infra00:57
mordredjeblair: wow00:57
mordredjeblair: nicely done!00:58
jeblairso that's a simple commit on top of the 2.5.0 tag; i'm working on a pull-request for master, but master is broken00:58
mordredjeblair: wow. that's great00:59
*** sarob_ has joined #openstack-infra00:59
clarkbI have officially given up on xfce's monitor management. xrandr ftw00:59
mordredand people wonder why we like gating...00:59
mordredclarkb: I actually use arandr00:59
mordredclarkb: which I find quite lovely when I'm doing external monitor/projector and placement and whatnot01:00
mordredalso, it's in python :)01:00
*** CaptTofu has quit IRC01:02
*** CaptTofu has joined #openstack-infra01:02
*** sarob has quit IRC01:02
clarkbis it just an xrandr wrapper?01:03
*** sarob_ has quit IRC01:03
clarkboh it is a gui01:04
lifelessmordred: I'm presuming thats whats up, right?01:06
*** CaptTofu has quit IRC01:07
*** jerryz has quit IRC01:07
*** dcramer_ has joined #openstack-infra01:11
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Check devstack-gate changes on stable branch jobs
*** guohliu has quit IRC01:13
fungii've been perfectly happy with the xrandr cli for multiple/external monitors and projectors, relative placement, rotation, scaling, et cetera01:14
*** CaptTofu has joined #openstack-infra01:15
*** DennyZhang has joined #openstack-infra01:16
*** melwitt1 has quit IRC01:18
*** fifieldt has joined #openstack-infra01:22
openstackgerritA change was merged to openstack-infra/devstack-gate: Call a script to check for ERRORs in logs on tempest success
*** Ryan_Lane has quit IRC01:34
*** Ryan_Lane has joined #openstack-infra01:35
openstackgerritA change was merged to openstack-dev/pbr: Get rid of PyPI URL override in integration test
*** Ryan_Lane has quit IRC01:42
*** adalbas has quit IRC01:44
*** gyee has quit IRC01:47
*** xchu has joined #openstack-infra01:49
*** ryanpetrello has joined #openstack-infra01:52
*** DennyZhang has quit IRC01:54
*** wenlock has joined #openstack-infra02:01
*** guohliu has joined #openstack-infra02:11
*** scairbus has quit IRC02:18
*** yaguang has joined #openstack-infra02:27
*** yaguang has quit IRC02:28
*** yaguang has joined #openstack-infra02:28
*** yaguang has quit IRC02:29
*** yaguang has joined #openstack-infra02:29
*** DennyZhang has joined #openstack-infra02:30
*** emagana has quit IRC02:30
*** emagana has joined #openstack-infra02:31
mordredpasses now02:33
*** ryanpetrello has quit IRC02:33
clarkbmordred: which change made it pass?02:34
*** ryanpetrello has joined #openstack-infra02:35
*** rfolco has joined #openstack-infra02:35
*** emagana has quit IRC02:35
clarkbinteresting, I didn't expect that to fix it because d-g should be doing the same thing right?02:36
clarkbmordred: or was it unsetting the mirror stuff when run in before?02:39
mordredI think it was doing wrong things02:39
mordredor different things02:39
*** sarob has joined #openstack-infra02:46
*** sarob has quit IRC02:48
*** sarob has joined #openstack-infra02:49
*** sarob_ has joined #openstack-infra02:51
*** sarob has quit IRC02:53
*** sarob_ has quit IRC02:59
*** sarob has joined #openstack-infra03:00
*** sarob has quit IRC03:04
*** oubiwann_ has joined #openstack-infra03:05
*** rfolco has quit IRC03:07
*** openstack has joined #openstack-infra03:23
*** sets mode: +ns 03:23
*** sets mode: -o openstack03:29 *** Notice -- TS for #openstack-infra changed from 1381289018 to 131551703203:29
*** sets mode: +ct-s 03:29
*** fungi has joined #openstack-infra03:29
*** changbl has joined #openstack-infra03:29
*** torgomatic has joined #openstack-infra03:29
*** chmouel_ has joined #openstack-infra03:29
*** jd__ has joined #openstack-infra03:29
*** yaguang has joined #openstack-infra03:29
*** mancdaz_ has joined #openstack-infra03:29
*** mrodden1 has joined #openstack-infra03:29
*** danger_f` has joined #openstack-infra03:29
*** oubiwann_ has joined #openstack-infra03:29
*** ryanpetrello has joined #openstack-infra03:29
*** DennyZhang has joined #openstack-infra03:29
*** guohliu has joined #openstack-infra03:29
*** wenlock has joined #openstack-infra03:29
*** xchu has joined #openstack-infra03:29
*** fifieldt has joined #openstack-infra03:29
*** CaptTofu has joined #openstack-infra03:29
*** dcramer_ has joined #openstack-infra03:29
*** erfanian has joined #openstack-infra03:29
*** jhesketh__ has joined #openstack-infra03:29
*** matsuhashi has joined #openstack-infra03:29
*** mdenny has joined #openstack-infra03:29
*** odyi has joined #openstack-infra03:29
*** jhesketh has joined #openstack-infra03:29
*** ianw has joined #openstack-infra03:29
*** freyes has joined #openstack-infra03:29
*** sdake_ has joined #openstack-infra03:29
*** harlowja has joined #openstack-infra03:29
*** salv-orlando has joined #openstack-infra03:29
*** moted has joined #openstack-infra03:29
*** amotoki has joined #openstack-infra03:29
*** dguitarbite has joined #openstack-infra03:29
*** tian has joined #openstack-infra03:29
*** ArxCruz has joined #openstack-infra03:29
*** enikanorov-w has joined #openstack-infra03:29
*** Alex_Gaynor has joined #openstack-infra03:29
*** mkoderer has joined #openstack-infra03:29
*** che-arne has joined #openstack-infra03:29
*** jesusaurus has joined #openstack-infra03:29
*** basic` has joined #openstack-infra03:29
*** jaypipes has joined #openstack-infra03:29
*** primemin1sterp has joined #openstack-infra03:29
*** retr0h has joined #openstack-infra03:29
*** nijaba_ has joined #openstack-infra03:29
*** sdake has joined #openstack-infra03:29
*** HenryG has joined #openstack-infra03:29
*** vipul has joined #openstack-infra03:29
*** openstackgerrit has joined #openstack-infra03:29
*** slong has joined #openstack-infra03:29
*** sileht has joined #openstack-infra03:29
*** olaph has joined #openstack-infra03:29
*** notmyname has joined #openstack-infra03:29
*** portante has joined #openstack-infra03:29
*** kiall_ has joined #openstack-infra03:29
*** locke1051 has joined #openstack-infra03:29
*** kpepple has joined #openstack-infra03:29
*** flaper87|afk has joined #openstack-infra03:29
*** pleia2 has joined #openstack-infra03:29
*** dstufft has joined #openstack-infra03:29
*** rwsu has joined #openstack-infra03:29
*** cody-somerville has joined #openstack-infra03:29
*** mgagne has joined #openstack-infra03:29
*** briancline has joined #openstack-infra03:29
*** alaski has joined #openstack-infra03:29
*** marun has joined #openstack-infra03:29
*** dripton has joined #openstack-infra03:29
*** markmcclain has joined #openstack-infra03:29
*** bnemec has joined #openstack-infra03:29
*** Shrews has joined #openstack-infra03:29
*** AJaeger has joined #openstack-infra03:29
*** mkerrin has joined #openstack-infra03:29
*** dkehn has joined #openstack-infra03:29
*** saper has joined #openstack-infra03:29
*** hemnafk has joined #openstack-infra03:29
*** klrmn has joined #openstack-infra03:29
*** oubiwann has joined #openstack-infra03:29
*** krtaylor has joined #openstack-infra03:29
*** mindjiver has joined #openstack-infra03:29
*** david-lyle has joined #openstack-infra03:29
*** sirushti has joined #openstack-infra03:29
*** mtreinish has joined #openstack-infra03:29
*** jomara has joined #openstack-infra03:29
*** jeblair has joined #openstack-infra03:29
*** uvirtbot has joined #openstack-infra03:29
*** Adri2000 has joined #openstack-infra03:29
*** dtroyer has joined #openstack-infra03:29
*** EmilienM has joined #openstack-infra03:29
*** Guest74104 has joined #openstack-infra03:29
*** adam_g has joined #openstack-infra03:29
*** ttx has joined #openstack-infra03:29
*** GheRivero has joined #openstack-infra03:29
*** niska` has joined #openstack-infra03:29
*** fbo_away has joined #openstack-infra03:29
*** annegentle_ has joined #openstack-infra03:29
*** jeremyb has joined #openstack-infra03:29
*** beagles has joined #openstack-infra03:29
*** guitarzan has joined #openstack-infra03:29
*** morganfainberg has joined #openstack-infra03:29
*** ekarlso has joined #openstack-infra03:29
*** mikal has joined #openstack-infra03:29
*** jswarren has joined #openstack-infra03:29
*** Reapster has joined #openstack-infra03:29
*** belliott has joined #openstack-infra03:29
*** rahmu has joined #openstack-infra03:29
*** zigo_ has joined #openstack-infra03:29
*** stevebaker has joined #openstack-infra03:29
*** SpamapS has joined #openstack-infra03:29
*** Protux has joined #openstack-infra03:29
*** rickerc has joined #openstack-infra03:29
*** mkirk has joined #openstack-infra03:29
*** jtzl has joined #openstack-infra03:29
*** cyeoh has joined #openstack-infra03:29
*** lifeless has joined #openstack-infra03:29
*** justinabrahms has joined #openstack-infra03:29
*** samalba has joined #openstack-infra03:29
*** dhellmann has joined #openstack-infra03:29
*** antonym has joined #openstack-infra03:29
*** richardwoo_ has joined #openstack-infra03:29
*** akscram has joined #openstack-infra03:29
*** gaelL has joined #openstack-infra03:29
*** sdague has joined #openstack-infra03:29
*** thingee_zzz has joined #openstack-infra03:29
*** saschpe has joined #openstack-infra03:29
*** soren has joined #openstack-infra03:29
*** LinuxJedi has joined #openstack-infra03:29
*** koolhead17 has joined #openstack-infra03:29
*** EntropyWorks has joined #openstack-infra03:29
*** paul-- has joined #openstack-infra03:29
*** rainya has joined #openstack-infra03:29
*** dragondm has joined #openstack-infra03:29
*** bodepd has joined #openstack-infra03:29
*** Daviey has joined #openstack-infra03:29
*** zehicle has joined #openstack-infra03:29
*** mordred has joined #openstack-infra03:29
*** jog0 has joined #openstack-infra03:29
*** Vivek has joined #openstack-infra03:29
*** dansmith has joined #openstack-infra03:29
*** westmaas has joined #openstack-infra03:29
*** Mithrandir has joined #openstack-infra03:29
*** clayg has joined #openstack-infra03:29
*** rockstar has joined #openstack-infra03:29
*** jcooley has joined #openstack-infra03:29
*** jgriffith has joined #openstack-infra03:29
*** russellb has joined #openstack-infra03:29
*** arosen has joined #openstack-infra03:29
*** jmeridth has joined #openstack-infra03:29
*** vishy has joined #openstack-infra03:29
*** NobodyCam has joined #openstack-infra03:29
*** lyz has joined #openstack-infra03:29
*** matty_dubs|gone has joined #openstack-infra03:29
*** hub_cap has joined #openstack-infra03:29
*** SlickNik has joined #openstack-infra03:29
*** plomakin_ has joined #openstack-infra03:29
*** wayneeseguin has joined #openstack-infra03:29
*** sbadia has joined #openstack-infra03:29
*** adarazs has joined #openstack-infra03:29
*** davidlenwell has joined #openstack-infra03:29
*** Ng has joined #openstack-infra03:29
*** BobBall has joined #openstack-infra03:29
*** mugsie has joined #openstack-infra03:29
*** dachary has joined #openstack-infra03:29
*** jlk has joined #openstack-infra03:29
*** zaro has joined #openstack-infra03:29
*** swifterdarrell has joined #openstack-infra03:29
*** clarkb has joined #openstack-infra03:29
*** simonmcc has joined #openstack-infra03:29
*** spiffxp has joined #openstack-infra03:29
*** ftcjeff__ has joined #openstack-infra03:29
*** maurosr has joined #openstack-infra03:29
*** juice has joined #openstack-infra03:29
*** psedlak has joined #openstack-infra03:29
*** wannatest has joined #openstack-infra03:29
*** devananda has joined #openstack-infra03:29
*** ChanServ has joined #openstack-infra03:29
*** sets mode: +o ChanServ03:29
*** changes topic to "Discussion of OpenStack Project Infrastructure | Docs | Bugs | Code"03:29
lifelessmordred: clarkb: so - can I have +2's ?
lifelessand up03:34
*** erfanian has quit IRC03:37
clarkblifeless: done03:39
*** afazekas has joined #openstack-infra03:41
*** dguitarbite has quit IRC03:50
*** dguitarbite has joined #openstack-infra03:51
*** emagana has joined #openstack-infra03:57
*** dguitarbite has left #openstack-infra03:59
*** afazekas has quit IRC04:01
*** dguitarbite has joined #openstack-infra04:05
*** sarob has joined #openstack-infra04:07
*** ArxCruz has quit IRC04:12
*** ArxCruz has joined #openstack-infra04:14
*** sarob has quit IRC04:16
*** sarob has joined #openstack-infra04:16
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Raise exceptions when no entrypoint, macro or template found
*** sarob has quit IRC04:21
*** yaguang has quit IRC04:23
*** yaguang has joined #openstack-infra04:23
*** yaguang has quit IRC04:28
*** yaguang has joined #openstack-infra04:34
*** afazekas has joined #openstack-infra04:38
openstackgerritA change was merged to openstack-dev/pbr: Fix test_changelog when git isn't globally setup.
*** wenlock has quit IRC04:49
*** afazekas has quit IRC04:53
*** scairbus has joined #openstack-infra04:56
*** scairbus has quit IRC04:58
*** scairbus has joined #openstack-infra04:59
*** scairbus has quit IRC05:00
*** CaptTofu has quit IRC05:02
*** CaptTofu has joined #openstack-infra05:02
*** boris-42 has joined #openstack-infra05:03
*** afazekas has joined #openstack-infra05:07
openstackgerritMerlijn van Deen proposed a change to openstack-infra/git-review: Split git rev-parse --show-toplevel --git-dir on newline
*** DennyZhang has quit IRC05:25
*** sarob has joined #openstack-infra05:27
*** SergeyLukjanov has joined #openstack-infra05:27
*** sarob has quit IRC05:32
*** oubiwann_ has quit IRC05:37
*** yaguang has quit IRC05:38
*** yaguang has joined #openstack-infra05:39
openstackgerritRoman Podolyaka proposed a change to openstack-infra/config: Use script for sqlalchemy-migrate jobs
*** SergeyLukjanov has quit IRC06:02
*** dguitarbite has quit IRC06:09
*** ryanpetrello has quit IRC06:14
*** xchu has quit IRC06:19
*** odyssey4me4 has joined #openstack-infra06:20
*** mindjive1 has joined #openstack-infra06:22
*** dstufft_ has joined #openstack-infra06:23
*** xchu has joined #openstack-infra06:23
*** dkehn_ has joined #openstack-infra06:25
*** bnemec has quit IRC06:25
*** SergeyLukjanov has joined #openstack-infra06:25
*** vipul is now known as vipul-away06:25
*** Ryan_Lane has joined #openstack-infra06:26
*** dstufft has quit IRC06:26
*** bnemec has joined #openstack-infra06:26
*** mindjiver has quit IRC06:26
*** dkehn has quit IRC06:26
*** marun has quit IRC06:36
*** SergeyLukjanov has quit IRC06:37
*** bnemec has quit IRC06:37
*** bnemec_ has joined #openstack-infra06:37
*** pblaho has joined #openstack-infra06:44
*** flaper87|afk is now known as flaper8706:46
*** SergeyLukjanov has joined #openstack-infra06:46
sorenmordred: You got it.06:49
*** reed has joined #openstack-infra06:49
sorenuvirtbot: config supybot.reply.whenAddressedBy.chars #06:49
uvirtbotsoren: The operation succeeded.06:49
sorenmordred: ^06:49
*** SergeyLukjanov has quit IRC06:53
*** Shrews has quit IRC07:00
*** odyssey4me4 is now known as odyssey4me07:00
*** Shrews has joined #openstack-infra07:00
reedttx, you there?07:00
*** vipul-away is now known as vipul07:01
*** AJaeger has quit IRC07:01
*** AJaeger has joined #openstack-infra07:01
openstackgerritHaomeng,Wang proposed a change to openstack/requirements: Set python-six minimum version
*** hemnafk has quit IRC07:11
*** mdenny has quit IRC07:12
*** LinuxJedi has quit IRC07:12
*** mdenny has joined #openstack-infra07:12
*** hemnafk has joined #openstack-infra07:12
*** SlickNik has quit IRC07:13
*** osanchez has joined #openstack-infra07:13
*** jesusaurus has quit IRC07:13
*** zaro has quit IRC07:14
*** GheRivero has quit IRC07:14
*** osanchez is now known as OlivierSanchez07:14
*** danger_f` has quit IRC07:15
*** SlickNik has joined #openstack-infra07:15
*** danger_fo_away has joined #openstack-infra07:16
*** jesusaurus has joined #openstack-infra07:17
*** zaro has joined #openstack-infra07:17
*** GheRivero has joined #openstack-infra07:20
openstackgerritHaomeng,Wang proposed a change to openstack/requirements: Set python-six minimum version
*** flaper87 is now known as flaper87|afk07:24
*** david-lyle has quit IRC07:26
*** david-lyle has joined #openstack-infra07:28
*** jcoufal has joined #openstack-infra07:30
*** LinuxJedi has joined #openstack-infra07:32
*** shardy has joined #openstack-infra07:37
*** dansmith has quit IRC07:38
openstackgerritEkaterina Fedorova proposed a change to openstack-infra/config: Add murano-repository to stackforge
*** dansmith has joined #openstack-infra07:43
*** sshturm has joined #openstack-infra07:44
*** rpodolyaka has joined #openstack-infra07:44
*** sdake has quit IRC07:45
*** sdake_ has quit IRC07:45
*** AJaeger has left #openstack-infra07:46
*** dizquierdo has joined #openstack-infra07:46
*** hashar has joined #openstack-infra07:50
*** markmc has joined #openstack-infra07:53
*** Ryan_Lane has quit IRC07:54
*** yassine has joined #openstack-infra07:57
ttxreed: yes08:00
reedttx, now I don't remember what it was about08:01
ttxha ha08:03
ttxmordred: feel free to hit me up with that "extra other thing to do on release day" explanation today08:04
ttxI'm all ears08:04
*** boris-42 has quit IRC08:06
*** flaper87|afk is now known as flaper8708:06
*** jpich has joined #openstack-infra08:07
reedttx, when is the release? (this is not the question I had for you, still trying to remember)08:16
ttxreed: next Thursday08:16
*** derekh has joined #openstack-infra08:18
*** fbo_away is now known as fbo08:21
*** sdake has joined #openstack-infra08:23
*** sdake has joined #openstack-infra08:23
sshturmclarkb:  hello, i'm adding a testing of migrations to oslo and for success i need another permissions for mysql user `openstack_citest` (it should be able to create temporary database).08:26
sshturmclarkb: Here is my patch and it is failed only on mysql. More than there is another patch with similar problems. Could you help on that?08:26
reedttx, do you have jeblair's public gpg key?08:30
* ttx checks08:31
ttxnot in my web of trust. But that would be
sshturmclarkb: In many components of OpenStack there are tests using one database. In concurrency mode we can get a lot of problems with synchronizing.  Decision of this problem by temporary database can cut testing time (every test will use it's own database).08:33
reedah, thanks ttx, he didn't add identity08:34
ttxreed: actually I should be able to verify it we have a few people in common08:35
sshturmmordred: What do you think about my previous messages to clarkb?  How can we make it possible?08:35
*** yaguang has quit IRC08:36
*** sdake has quit IRC08:36
*** sdake has joined #openstack-infra08:43
ttxreed: looks pretty valid to me but you should not trust me :)08:43
reednever trust anybody08:44
ttxespecially on irc08:44
*** dizquierdo has quit IRC08:45
*** jhesketh__ has quit IRC08:48
*** jhesketh__ has joined #openstack-infra08:48
*** sdake has quit IRC08:49
notmynamettx: looks like I've overflowed and looped back to zero and am still awake when you woke up. are we good for swift? ('cause I really don't want to wake up in 4 hours to check on it ;-)08:56
notmynamettx: ah. looking at my email from LP, I'm guessing you're on it now :-)08:57
koolhead17reed: are you working from Italy?08:58
koolhead17its too late for everyone in US AFAIK :D09:00
notmynamekoolhead17: hear hear09:00
koolhead17notmyname: hahaha09:01
notmynamettx: I'm going to bed. I'll check email/IRC in 6-ish hours09:01
ttxnotmyname: on it09:02
notmynamettx: kk. need anything from me?09:03
ttxnotmyname: all set, just working on pushing tag and publishing tarball09:03
ttxnotmyname: I can announce the RC1 when done09:03
notmynamettx: ok, thanks09:03
ttxtag pushed, let's see if pbr works09:04
notmynamettx: I'll get the release notes for the full havana release done on the wiki as soon as I can09:04
ttxTarball generated at swift-1.10.0.rc1.tar.gz09:04
notmynamettx: my message for the RC is that it is 1.10.0, and havana includes 1.9.0, 1.9.1, and 1.10.009:05
notmynamettx: wrt the LP tracked stuff09:05
ttxok, will mention that09:05
ttxnotmyname: you should sleep now :)09:05
notmynamettx: heh :-)09:05
notmynamewill do09:06
ttxall up on
*** sdake has joined #openstack-infra09:09
*** mrmartin has joined #openstack-infra09:09
*** yaguang has joined #openstack-infra09:12
*** arosen has quit IRC09:12
ttxnotmyname: announced via email & twitter09:13
*** Bada has joined #openstack-infra09:15
*** Bada has quit IRC09:20
*** dizquierdo has joined #openstack-infra09:22
*** arosen has joined #openstack-infra09:24
*** sdake has quit IRC09:25
*** sdake has joined #openstack-infra09:27
*** sdake has quit IRC09:35
*** sdake has joined #openstack-infra09:36
*** SergeyLukjanov has joined #openstack-infra09:41
*** sdake has quit IRC09:41
*** basha has joined #openstack-infra09:46
*** jhesketh has quit IRC09:53
hasharjeblair: I get some nice graphs using graphite :-D thx!
*** xchu has quit IRC10:07
ttxmordred: more stupidity:
*** sdake has joined #openstack-infra10:11
*** sdake has joined #openstack-infra10:11
hasharmeanwhile @ wikimedia 1/3 of eng folks are remotees10:12
*** yaguang has quit IRC10:12
ttxmeanwhile @ $smarter_companies we understand that the right people matter more than middle management control freaks10:13
hasharthat is challenging though10:13
hasharthe remote might be missing part of the engagement / goals set up by directors10:13
hasharbut then we are probably more productive since we escape the offices drama10:14
hasharit is hard really10:14
ttxhashar: challenging yes... make sit harder for middle management to justify their lunch expense reports10:14
hasharwhat I would love to see is small coworking hubs managed by big companies where people can hang out instead of commuting10:14
ttxsince they don't have a glassed office in the openspace corner10:15
ttxhashar: I like the quote from that article though10:15
ttx"We made stupid expenses in improving our offices so now better have people to fill them.10:16
*** fbo is now known as fbo_away10:16
hasharsome business analytic dashboard must have shown that the earning per square foot was not good enough so they get more people there to raise the ratio and make the accountants happy / dashboard green again10:17
hashar""the construction of a nursery near her office at Yahoo HQ ""10:17
hasharthat is handy10:17
*** flaper87 is now known as flaper87|afk10:17
*** alcabrera has joined #openstack-infra10:27
sdagueit is interesting so see so many companies shift that way, sounds like a good beer topic for trying to figure out why :)10:28
*** afazekas has quit IRC10:30
sdaguehashar: I think it's more challenging in the situation where you are in where remotes don't dominate. All my projects over the past 10 years we rarely have even as much 30% of the people at the same location. Which makes everyone a remote.10:31
*** pycabrera has joined #openstack-infra10:32
*** pcm_ has joined #openstack-infra10:33
*** pcm_ has quit IRC10:34
*** pcm_ has joined #openstack-infra10:34
*** alcabrera has quit IRC10:35
*** DinaBelova has joined #openstack-infra10:39
hasharsdague: maybe some people abused their position10:43
hasharalso it might be hard to enroll remotes people in the enterprise culture/brainwashing10:44
hasharat wikimedia, the whole story started with everyone being remote and all of our volunteers are remotes anyway. So that is part of our culture.10:44
hasharbut yeah, beer needed.10:44
sdagueI'm sure some people abused it, but people can just as easily shop on amazon from their cubicle as well :)10:45
*** dstufft_ is now known as dstufft10:50
*** boris-42 has joined #openstack-infra10:55
*** adalbas has joined #openstack-infra10:55
*** pblaho has quit IRC10:57
*** pblaho has joined #openstack-infra11:00
*** guohliu has quit IRC11:01
Mithrandirhmm, I wonder if we should extend the jjb yaml slightly with support for !include11:02
openstackgerritA change was merged to openstack-dev/pbr: Switch away from tearDown for BaseTests.
*** alexpilotti has joined #openstack-infra11:07
*** zul has joined #openstack-infra11:14
*** enikanorov-w_ has joined #openstack-infra11:15
*** enikanorov-w has quit IRC11:17
*** flaper87|afk is now known as flaper8711:18
*** pycabrera is now known as alcabrera11:18
*** weshay has joined #openstack-infra11:18
*** emagana has quit IRC11:20
*** emagana has joined #openstack-infra11:20
*** emagana has quit IRC11:25
*** afazekas has joined #openstack-infra11:25
*** hashar has quit IRC11:26
*** alexpilotti_ has joined #openstack-infra11:29
*** hashar has joined #openstack-infra11:30
*** Bada has joined #openstack-infra11:30
*** alexpilotti has quit IRC11:32
*** alexpilotti_ is now known as alexpilotti11:32
*** Bada has quit IRC11:35
*** fbo_away is now known as fbo11:36
*** DinaBelova has quit IRC11:39
*** dcramer_ has quit IRC11:40
*** CaptTofu has quit IRC11:45
*** CaptTofu has joined #openstack-infra11:45
*** salv-orlando has quit IRC11:51
sdagueit's amazing how smooth the gate runs when the race conditions in the projects get fixed11:52
*** DinaBelova has joined #openstack-infra11:54
hasharsomeone from Google checked in my coworking place11:57
hasharguess they still have remote workers11:57
*** dprince has joined #openstack-infra11:58
*** rfolco has joined #openstack-infra12:07
*** lcestari has joined #openstack-infra12:08
*** Bada has joined #openstack-infra12:11
*** Bada has quit IRC12:16
*** markmcclain has quit IRC12:18
*** OlivierSanchez has quit IRC12:19
*** osanchez has joined #openstack-infra12:24
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: statsd documentation
*** sandywalsh has joined #openstack-infra12:30
mordredttx: yes. I'm working on that with the management chain and whatnot12:32
*** CaptTofu has quit IRC12:34
*** CaptTofu has joined #openstack-infra12:34
*** hashar has quit IRC12:36
*** odyssey4me has quit IRC12:37
ttxmordred: -ETOOMANYGOODFIGHTS12:42
*** dizquierdo has quit IRC12:43
sshturmmordred: hello, i'm adding a testing of migrations to oslo and for success i need another permissions for mysql user `openstack_citest` (it should be able to create temporary database)12:44
sshturmmordred:  Here is my patch and it is failed only on mysql. More than there is another patch with similar problems. Could you help on that?12:44
mordredsshturm: can you explain "create temporary database" ?12:44
sshturmmordred: In many components of OpenStack there are tests using one database. In concurrency mode we can get a lot of problems with synchronizing.  Decision of this problem by temporary database can cut testing time (every test will use it's own database).12:45
*** fifieldt has quit IRC12:45
sshturmmordred: i mean that we can generate database name for each test and run test in it's own database instead of using one database for all tests.12:46
*** jomara has quit IRC12:47
*** dkranz has joined #openstack-infra12:49
sshturmmordred:  looks like for mysql user `openstack_citest` was created only  for openstack_citest database, but it will be better to have access to arbitrary database for this user12:50
*** fbo has quit IRC12:51
*** mrmartin has quit IRC12:51
*** fbo has joined #openstack-infra12:51
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add missed savanna-docs job to post pipeline
mordredah. gotcha.12:53
mordredsshturm: let me think a bit about a good way to solve that12:53
*** Bada has joined #openstack-infra12:53
openstackgerritChristian Berendt proposed a change to openstack/requirements: six has to be in version 1.4.0 or higher
sshturmmordred: without these permissions we should use lockutils for every test that will use shared database. But very often it is doesn't resolve this problem.12:53
*** sarob has joined #openstack-infra12:54
*** julim has joined #openstack-infra12:54
*** matty_dubs|gone is now known as matty_dubs12:54
sshturmmordred: this problem (limited permissions) we have only for mysql...12:55
*** BadaBoum has joined #openstack-infra12:56
*** Bada has quit IRC12:58
*** weshay has quit IRC12:59
*** pabelanger has joined #openstack-infra12:59
*** anteaya has joined #openstack-infra13:00
*** thomasm has joined #openstack-infra13:02
*** DennyZhang has joined #openstack-infra13:02
*** hashar has joined #openstack-infra13:03
*** alcabrera is now known as alcabrera|afk13:04
*** BadaBoum has quit IRC13:05
*** Bada has joined #openstack-infra13:05
*** jergerber has joined #openstack-infra13:08
*** jpeeler has joined #openstack-infra13:11
*** jpeeler has quit IRC13:11
*** jpeeler has joined #openstack-infra13:11
*** mriedem has joined #openstack-infra13:13
*** ryanpetrello has joined #openstack-infra13:14
ryanpetrelloping :)?13:14
*** jpeeler has left #openstack-infra13:15
sdaguesshturm: is there a reason to not just put locking around the db tests and use the single db?13:15
openstackgerritA change was merged to openstack/requirements: Set python-six minimum version
sdagueone of the advantages of the pre-specified user is that it lets people have control over their dev env for running unit tests with assurity that it won't nuke their system13:16
mordredjust for the record:13:17
mordred    # mysql::db is too dumb to realize that the same user can have13:17
mordred    # access to multiple databases and will fail if you try creating13:17
mordred    # a second DB with the same user. Create the DB directly as mysql::db13:17
mordred    # above is creating the user for us.13:17
mordredthat's in the puppet code around creating databases and is incorrect13:17
mordredthe puppet mysql::db module is too stupid to do grant all properly13:17
mordredand instead is being "clever"13:17
mordredand only granting all perms on databases that were created by puppet, and is not granting all at the global level13:18
mordreda) it's possible to do13:18
mordredb) we should have a conversation about whether or not it's how we want the slave db configured13:18
*** jpeeler has joined #openstack-infra13:19
dhellmanngood morning13:20
dhellmannlooks like there's an issue with the propose-requirements-updates job?
mordredmorning dhellmann13:20
mordreddhellmann: neat! thanks.13:20
dhellmannor maybe just that build system?13:20
mordrednope. it should happily handle that I believe13:21
*** changbl has quit IRC13:21
*** dcramer_ has joined #openstack-infra13:23
mordreddhellmann: yup. it's a bug13:24
dhellmannmordred: yay?13:24
*** weshay has joined #openstack-infra13:25
*** yassine has quit IRC13:28
*** matsuhashi has quit IRC13:30
*** blamar has joined #openstack-infra13:30
*** matsuhashi has joined #openstack-infra13:30
zuldhellmann:  say it with conviction...yay! :)13:30
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Clean up in requirements job
mordredfungi: ^^13:31
mordredfungi: double check me there?13:31
* fungi looks13:31
*** dkehn_ is now known as dkehn13:32
*** krtaylor has quit IRC13:32
mordreddhellmann: while I've got your attention: ?13:33
mordreddhellmann: (I'm checking your patch that depends on it right now)13:33
*** yassine has joined #openstack-infra13:34
*** matsuhashi has quit IRC13:34
openstackgerritA change was merged to openstack-infra/config: Clean up in requirements job
ekarlso < anyone care for signing off this one ?13:38
ekarlsobeen waiting for a loong time13:38
*** dizquierdo has joined #openstack-infra13:39
*** dafter has joined #openstack-infra13:40
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: use join to list multiple bugs
*** dcramer_ has quit IRC13:41
dhellmannmordred: looking13:43
dhellmannmordred: python -m pip.__init__ install? really? le sigh.13:44
*** blamar has quit IRC13:44
mordreddhellmann: yah man13:44
dhellmannI get it. Still.13:45
mordreddhellmann: the amazing cantrips I've had to come up with13:45
dhellmannmordred: backwards compatibility is a slippery slope13:46
*** pabelanger has quit IRC13:46
*** DinaBelova has quit IRC13:46
dhellmannmordred: pretty soon you'll be trying to write software that works most of the time13:46
ttxmordred: am available now if you want to talk about that extra task on release day13:46
sdaguedhellmann: well at least we have elastic recheck to start quantifying our work percentage :)13:46
ttxmordred: i'm not too grumpy yet13:47
sdagueit's funny that when software gets complex enough there is no longer binary  works/broken, but a percentage value13:47
*** thedodd has joined #openstack-infra13:52
*** pblaho has quit IRC13:55
mordredttx: it's should be pretty easy ish13:55
mordredttx: what we're adding is an openstack super-repo that contains all of the other repos as submodules13:56
mordredttx: the state of the submodules will be maintained by gerrit, such that a given commit in the super repo will always contain a collection of shas that have actually been tested by zuul13:56
mordredttx: (that's all automatic and we get it for free)13:56
*** markmcclain has joined #openstack-infra13:56
*** pblaho has joined #openstack-infra13:56
mordredttx: what _you_ may need to do (and by you, I mean the release team) are a couple of things13:56
mordredttx: one is, and I need to cook up process for this first, I believe that the super repo itself should be tagged once all of the subrepos have been tagged13:57
mordredttx: and the tag on the super repo should probably contain the tagged versions of each subrepo (so that checking out 2013.2 of the super repo and then doing a submodule update will get you what you think it will)13:57
mordredttx: secondly, we'll want to make a stable/X branch of the super repo, so that updates to the stable branches will be tracked13:58
ttxmordred: hmmkay13:58
ttxmordred: why would you do that super-repo in the first place ?13:58
mordredttx: thirdly, when we add new projects to the integrated release, we'll want to add them to the super repo13:58
mordredttx: the main problem being solved13:58
mordredttx: is for people doing CD-deployments, making sure that they can find a programatically accessible record of which sets of commits go together13:59
mordredmaster is tested with master, but you don't know which commit in a given branch was master at the time a different commit in a different repo was master13:59
ttxI see13:59
ttxmordred: well you could use dates, but that's a bit painful13:59
mordredgerrit has a feature called submodule subscription which makes the superrepo very easy to maintain in an ongoing fashion (it updates the submodules every time a commit lands to one of them automatically)14:00
ttxi.e. master at a given date is always tested14:00
*** dims has joined #openstack-infra14:00
*** yassine has quit IRC14:00
ttxmordred: would it change anything from the submodule (our current projects/repo) perspective ?14:01
*** yassine has joined #openstack-infra14:01
*** hashar is now known as hasharCall14:01
*** oubiwann_ has joined #openstack-infra14:02
mordrednot at all14:03
ttxmordred: question 2: about creating stable/* on the superrepo: when would you do that ? just after all the milestone-proposed branches get migrated to stable/* ?14:03
mordredmost people need never know anything about this repo14:03
mordredttx: yeah. I think you want all the branches in the submodules that are going to exist to exist properly14:03
ttxmordred: ok, so that would all be "postrelease" tasks in the general release process (as described in
mordredttx: and then there is a thing I can show you that will need to be edited by hand for now - which will tell gerrit to grab master for the client libs, for instance14:04
ttxthen I'm fine, postrelease is not time-bound14:04
*** basha has quit IRC14:04
ttx(i.e. can actually happen in the hours AFTER the announced release date)14:05
ttx(I usually do postrelease on the Friday)14:05
ttxmordred: so I don't think pushing buttons for those extra tasks would be a problem, as long as you script them in some way14:06
ttxfwiw almost everything is scripted now, as this page shows14:07
mordredttx: I will script them - next week may not be scripted so much yet14:07
mordredttx: but I'll help with that more explicitly, which will then help to sort out scripting14:07
ttxmordred: btw did we get to a gerrit version that can support branch creation scripts ?14:07
mordrednot yet14:07
mordredwe're much closer though14:07
ttxthat's like the only part I can't script at this point :)14:07
*** dkranz has quit IRC14:08
ttxthe biggest pain being scripts that timeout while trying to change bugtask status in Launchpad for bugs with more than 4 tasks. Thanks lifeless for crazy API timeouts :)14:09
ttxthat's become my main incentive for moving off LP.14:09
ttx(and before anyone asks, yes, there is a bug opened for that, bug 1076512)14:13
uvirtbotLaunchpad bug 1076512 in launchpad "BugTask:+editstatus timeouts on bugs with many tasks" [Critical,Triaged]
*** Bada has quit IRC14:13
*** CaptTofu has quit IRC14:16
*** CaptTofu has joined #openstack-infra14:16
*** pabelanger has joined #openstack-infra14:17
*** blamar has joined #openstack-infra14:19
mordredttx: nod14:23
mordredttx: I'm TRYING REAL HARD to get you some storyboard devs14:23
ttxsdague: either elastic-recheck is smarter than me or it points to the wrong bug14:24
fungithere's at least one bug i've been trying to update for a while, but it has so many bugtasks on it that the lp interface times out on any status change operations... maddening14:24
ttxfungi: the funny workaround is to use the email interface14:24
anteayattx he mentions it to me almost every meeting14:24
fungioh! will try that14:24
anteayaand I am keen to help14:24
*** dkranz has joined #openstack-infra14:24
fungittx: thanks!!! bookmarked14:25
anteayabut I lack the understanding around what is actually happening with releases to feel that I know what the code is trying to accomplish14:25
anteayabefore I have that, I feel like I am just doing busywork without a goal14:25
anteayaI dislike busywork14:25
anteayattx so to that end, can I book some of your time at the summit14:25
anteayato help me understand the release cycle and its nuances in more depth14:26
anteayasince all I have right now is "and then ttx waves his magic wand"14:26
anteayawhich is hard to code14:26
ttxsdague: it's actually smarter than me.14:26
anteayaor at least is hard for me to code14:26
ttxsdague: or at least , more thorough.14:26
*** blamar has quit IRC14:27
annegentle_ttx: and mordred: and infra peeps, we're looking at ways to make sure OPW interns have good work that's sort of "reserved" for them since it is tough to find 3-month bite-sized projects.14:27
mordredannegentle_: I think that's going to be tough14:27
annegentle_ttx: mordred: jeblair: others -- would it be okay to assign them to potential mentors as a placeholder? Or will that be a horrible idea?14:27
mordredannegentle_: is the antecedant to them in that sentence "the tasks" ?14:27
annegentle_mordred: it's hard enough to find meaningful projects for interns, the nice thing about OPW is it's not just students14:28
annegentle_mordred: heh, "them" would be a blueprint14:28
mordredI think that assigning a blueprint to a mentor as a placeholder is just fine14:28
annegentle_mordred: so I'm wondering if "parking" on a blueprint is poor form and will cause us to be poked fun at :)14:28
mordredI think that once the intern exists, the blueprint needs to be transferred to them14:29
annegentle_mordred: NICE. thanks. I can't think of it being a real blocker to others, but it sure helps us look for good work14:29
annegentle_mordred: yep, that'd be the intent14:29
mordredbut - if someone pokes a mentor and says 'omg, I need that done quickly, can I take it from you' - I think you need to be open to that :)14:29
ttxsdague: btw you may have some "closed" bugs still triggering occurences due to milestone-proposed not containing the right fixes yet14:29
mordredcause, you know, cats14:29
*** sarob has quit IRC14:30
*** SergeyLukjanov has quit IRC14:30
*** blamar has joined #openstack-infra14:30
*** sarob has joined #openstack-infra14:30
*** salv-orlando has joined #openstack-infra14:32
*** ArxCruz has quit IRC14:33
*** salv-orlando has quit IRC14:33
*** changbl has joined #openstack-infra14:34
annegentle_mordred: oh yes, it's a good lesson for mentor and intern, we had 2 of the 3 first interns get work taken out from under them unexpectedly, so that was another learning lesson in open source in itself14:34
dimsfungi,folks, rechecks and zuul pages are missing their css (example -
mordredannegentle_: ++14:35
*** sarob has quit IRC14:35
*** julim has quit IRC14:35
mordreddims: when you says missing their css, can you be more explicit?14:35
mordredoh - seems ugly for sure14:35
*** DennyZha` has joined #openstack-infra14:36
dimsmordred, y. rechecks page too14:36
mordredfungi: do you know how to ping todd?14:37
mordredfungi: apparently someone is doing something14:37
anteayaI think the important part in the exercise of changing assignee in work items is communication14:37
anteayathat was my take away14:38
mordredjeblair: ^^ ping if you're up14:38
mordredjeblair, fungi: short version of story, has been taken completely offline, which a) is crazypants but b) actually has an impact on sites we serve since we get our css from there14:38
mordredttx: ^^ do you know how to ping todd?14:39
ttxmordred: apart from sending an email he won't answer to ?14:39
ttxmordred: we have a weekly meeting that he sometimes attends. That was yesterday.14:39
*** pentameter has joined #openstack-infra14:40
fungihe usually answers his e-mail when he wakes up. jonathan also is usually fairly responsive in situations like this14:40
*** julim has joined #openstack-infra14:40
fungibut i can also open a case with rackspace on it14:40
mordredit looks like its on purpose14:41
mordredthere is an openstack logo on the down page14:41
mordredI just private-twittered collier14:41
*** yaguang has joined #openstack-infra14:41
fungiyeah, rereading what you wrote, now i get it14:41
mordredI REALLY want that site moved14:41
fungii previously thought you meant ddos or something14:41
mordredI mean, 'down for maint'14:42
fungithat's swell14:42
*** datsun180b has joined #openstack-infra14:42
fungii don't recall anyone mentioning this was on the schedule14:42
*** dcramer_ has joined #openstack-infra14:43
*** dkranz has quit IRC14:43
fungihalf the staff were otherwise occupied yesterday and couldn't make the meeting14:43
mordredtweets and emails sent14:45
anteayaback up14:45
mordredok. it's back up14:45
*** rcleere has joined #openstack-infra14:46
*** CaptTofu has quit IRC14:47
dimsy much better now14:47
*** sdake_ has joined #openstack-infra14:47
*** sdake_ has quit IRC14:47
*** sdake_ has joined #openstack-infra14:47
mordredfollow up email sent14:47
*** sdake has quit IRC14:47
fungimakes me think that until we get the site on infra-managed systems and become involved with the site update process, it might be better for us to have a cron job pull the css bits to a system under our control and link our sites to that copy instead14:50
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: Do not pass unicode where byte strings are wanted
mordredfungi: could I bother you for a review on a pbr change: ? it fixes a known bug on windows, and I'd like to get close to being able to release to fix it14:51
*** kiall_ is now known as Kiall14:51
fungiyou bet14:52
*** DinaBelova has joined #openstack-infra14:52
*** roaet has joined #openstack-infra14:55
*** rnirmal has joined #openstack-infra14:55
*** wenlock has joined #openstack-infra14:56
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Move base test case logic out of
*** dkranz has joined #openstack-infra14:56
*** CaptTofu has joined #openstack-infra14:58
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Add the semver documentation
*** oubiwann_ has quit IRC15:00
*** pabelanger has quit IRC15:01
*** prad has joined #openstack-infra15:03
*** mrodden1 has quit IRC15:04
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Move base test case logic out of
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
*** pabelanger has joined #openstack-infra15:04
*** DinaBelova has quit IRC15:05
*** flaper87 is now known as flaper87|afk15:05
*** SergeyLukjanov has joined #openstack-infra15:06
*** ArxCruz has joined #openstack-infra15:06
*** pblaho has quit IRC15:06
*** DinaBelova has joined #openstack-infra15:07
fungii need to head out for a bit to run some errands and grab lunch, but will be back in a couple hours15:07
sdaguettx: yeh, elastic recheck is early, if you think it's got a bad hit, we should change the query15:07
jeblairfungi, mordred: actually, we should just make local copies of the css and not cron it -- that way we're protected from the main site css changing as well as disappearing15:09
mordredjeblair: do you think we should do that for all time, or just until we get the site into gerrit?15:10
jeblairfungi, mordred: and fwiw, was a surprise to me too15:10
jeblairmordred: hrm.  with gerrit we'd at least have notice; but i'd lean toward keeping them separate anyway -- then the update process is reversed -- if we see a css update pending, then we get to make sure it works with our sites and switch to it on our schedule15:11
jeblairmordred: also, small network speed benefits for people who visit our sites without visiting www.o.o first.15:12
*** krtaylor has joined #openstack-infra15:12
jeblairmordred: (in fact, i want to cache the fonts too -- pulling them from google is the single slowest thing about visiting the status site atm)15:12
mordredjeblair: I'm on board with all of that15:12
mordredjeblair: should we put the css into gerrit and then make a cron job that checks and if there are changes proposes them to our copies?15:13
jeblairmordred: neat idea15:13
jeblairmordred: i'm eating breakfast and then i'm going out for an all-morning errand; need anything from me atm?15:13
*** hasharCall is now known as hashar15:14
mordredjeblair: uhm. a sausage?15:15
*** blamar has quit IRC15:16
*** afazekas has quit IRC15:17
*** changbl has quit IRC15:17
*** DinaBelova has quit IRC15:18
*** jcoufal has quit IRC15:18
*** pblaho has joined #openstack-infra15:19
*** afazekas has joined #openstack-infra15:19
*** HenryG has quit IRC15:19
*** branen has joined #openstack-infra15:20
*** jecarey has joined #openstack-infra15:20
*** blamar has joined #openstack-infra15:23
*** ArxCruz has quit IRC15:28
*** sarob has joined #openstack-infra15:29
*** svarnau has joined #openstack-infra15:30
*** SergeyLukjanov has quit IRC15:34
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix pep8 errors in modules dir
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove launchpad_sync module
sdaguejog0: easy +2 -
*** flaper87|afk is now known as flaper8715:39
svarnauHi, I'm wondering why launchpad_sync is no longer used. Does something else take it's place?15:40
mordredsvarnau: we manage groups directly in gerrit now15:41
*** pblaho has quit IRC15:41
mordredsvarnau: have done for at least a good 6 months, possibly longer?15:41
mordredsvarnau: we're still using launchpad openid as a SSO source for now, but that just gets the user account into gerrit15:42
mordredsvarnau: the syncing of users from launchpad directly into the database behind the scenes broke constantly - and ultimately gave very little benefit15:42
*** emagana has joined #openstack-infra15:43
svarnauOkay, good to know.  In trying to set up our own instance, the expire old reviews script broke because launchpadlib was not installed...15:43
svarnauso we thought the fix was to pull int he module15:44
svarnaumordred: pull in the launchpad_sync module.15:44
*** markmcclain has quit IRC15:45
*** yassine has quit IRC15:45
svarnauI'm thinking now that there may have been a dangling dependency in there somewhere15:45
*** HenryG has joined #openstack-infra15:46
*** DennyZhang has quit IRC15:47
*** DennyZha` has quit IRC15:47
*** flaper87 is now known as flaper87|afk15:47
*** pabelanger has quit IRC15:52
*** ArxCruz has joined #openstack-infra15:53
mordredsvarnau: thanks! That just pointed out to me that we don't install launchpad lib in a place that we do use it15:54
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Fix pep8 errors in modules dir
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove launchpad_sync module
svarnaumordred: glad you found that spot. I was trying to find it.15:55
*** thingee_zzz is now known as thingee16:00
svarnauOn a different topic, how does review/test process work if there are dependent changes in different repos?16:00
openstackgerritA change was merged to openstack-dev/pbr: Add a test for command registration.
*** gyee has joined #openstack-infra16:00
openstackgerritA change was merged to openstack-dev/pbr: Rework run_shell_command
*** matty_dubs is now known as matty_dubs|lunch16:00
*** boris-42 has quit IRC16:01
*** boris-42 has joined #openstack-infra16:02
mordredsvarnau: we do not currently support dependent changes in different repos16:03
mordredsvarnau: there have been discussions in that direction, but nothing implemented16:03
mordredsvarnau: currently we solve it via sequencing16:04
svarnaumordred: Okay, so each change must be local to a repo and compatible with the current/future behavior in the other repo?16:05
svarnaumordred: Got it. Thanks!16:05
*** MIDENN_ has joined #openstack-infra16:05
*** mdenny has quit IRC16:06
*** anteaya has quit IRC16:07
*** chrisstreeter has joined #openstack-infra16:10
mordredfungi, clarkb: eek:
*** enikanorov-w_ has quit IRC16:16
mordredclarkb, fungi: try voting here ^^ devananda just pointed it out to me16:16
*** pabelanger has joined #openstack-infra16:17
*** enikanorov-w has joined #openstack-infra16:17
*** basha has joined #openstack-infra16:18
devanandamordred, clarkb, fungi -- apparently there are hidden drafts published /after/ the initial 2 non-draft revs16:19
devanandaUI doesn't seem to like that16:20
NobodyCamit was /my fault16:20
NobodyCamdarn up arrow16:20
*** fbo is now known as fbo_away16:22
*** boris-42 has quit IRC16:23
*** boris-42 has joined #openstack-infra16:23
*** basha_ has joined #openstack-infra16:24
*** emagana has quit IRC16:24
*** pabelanger has quit IRC16:24
*** emagana has joined #openstack-infra16:24
*** basha has quit IRC16:25
*** basha_ is now known as basha16:25
*** jpich has quit IRC16:26
*** svarnau has quit IRC16:28
*** svarnau has joined #openstack-infra16:28
*** hogepodge has joined #openstack-infra16:29
*** emagana has quit IRC16:29
*** osanchez has quit IRC16:29
clarkbdont use drafts :P16:35
mordredclarkb, fungi, jeblair: when you get a chance - the libra guys would like the ability to tag their things16:35
*** hogepodge has quit IRC16:37
*** hashar has quit IRC16:38
*** thingee is now known as thingee_zzz16:42
*** hogepodge has joined #openstack-infra16:43
*** moted has quit IRC16:43
*** jmeridth has quit IRC16:44
*** chmouel_ is now known as chmouel16:51
*** svarnau_ has joined #openstack-infra16:52
*** amotoki has quit IRC16:52
*** basha has quit IRC16:52
*** vogxn has joined #openstack-infra16:53
*** boris-42 has quit IRC16:54
*** SergeyLukjanov has joined #openstack-infra16:54
*** Ryan_Lane has joined #openstack-infra16:54
*** svarnau has quit IRC16:54
*** boris-42 has joined #openstack-infra16:55
*** DennyZhang has joined #openstack-infra16:56
*** rnirmal has quit IRC16:57
*** MIDENN_ has quit IRC16:58
*** boris-42 has quit IRC16:58
*** mdenny has joined #openstack-infra16:58
*** Kiall has quit IRC17:01
*** CaptTofu has quit IRC17:01
*** CaptTofu has joined #openstack-infra17:02
*** derekh has quit IRC17:02
*** pabelanger has joined #openstack-infra17:03
*** yaguang has quit IRC17:03
*** datsun180b_ has joined #openstack-infra17:04
*** basha has joined #openstack-infra17:04
clarkbmordred: sshturm: yeah postgres allows us to be pretty fine grained about permissions, but mysql doesn't17:05
clarkbat least via the puppet modules17:05
mordredclarkb: not true17:05
mordredclarkb: or, at least, puppet sucks at it17:05
mordredmysql is fine17:05
clarkbright the reason postgres works today is the puppet module allows us to specify those perms17:05
clarkbmysql doesn't and is a bit hackish from the puppet side17:05
jog0sdague: A+ed17:06
clarkb(postgres and mysql here refer to the puppet modules)17:06
clarkbthat said at the very least we can have it run a command to grant permissions to add and drop schemas17:06
sdaguejog0: thanks17:07
*** datsun180b has quit IRC17:07
*** datsun180b_ is now known as datsun180b17:07
openstackgerritA change was merged to openstack-infra/elastic-recheck: use join to list multiple bugs
pleia2re: tripleo initial testing merged at jeblair said this yesterday: < jeblair> pleia2: nodepool isn't auto-updated right now, so the nodepool portion of that won't immediately take effect17:08
pleia2what process is used for making this available for using?17:08
clarkbmordred: is that something that can be done iwthout potentially derping the system schemas?17:08
clarkbmordred: I think postgres we gave the user permissions to adds schemas which gives the user ownership of the schema which allows it to drop the schema. This protects the other DBs in the server17:08
*** basha has quit IRC17:09
clarkbpleia2: jeblair has puppet disabled on the nodepool host as a result of the failed nodepool gearman autoscaling17:09
pleia2clarkb: ah, gotcha17:09
clarkbpleia2: jeblair needs to get nodepool to a place where puppet can be started again (I believe he is close, one prereq change has been written, now just need time to apply it)17:10
pleia2clarkb: so that needs to be fixed before new changes will apply?17:10
pleia2ok, thank you17:10
*** datsun180b has quit IRC17:10
*** datsun180b has joined #openstack-infra17:10
jog0sdague: the bot will need to be kicked to get your change17:11
*** moted has joined #openstack-infra17:11
fungijeblair: mordred: make sure that server-side caching of google's fonts is not against their tos (or better still, we confirm we're linking proper free fonts)17:15
fungimordred: devananda: and yes, that's normally what you see when someone pushes a draft patchset after a non-draft patchset. nothing new to see here, move along ;)17:15
devanandafungi: ack. thanks17:16
*** sarob has quit IRC17:16
*** sarob has joined #openstack-infra17:16
sdaguejog0: are we running the bot under puppet?17:16
clarkbsdague: we are, but puppet doesn't trigger a service restart17:16
fungiclarkb: is the reason why puppet-mysql's permission granularity is teh sux because of the ancient version we're on?17:16
clarkbsdague: in general we have stopped using puppet for that because it often does it at unfortunate times. eg meetbot restarting in the middle of meetings17:17
sdagueclarkb: ok, that shouldn't be too hard to do, right? Maybe I'll take a look17:17
jog0clarkb: heh17:17
sdagueyeh, well recheckbot restart should be safer17:17
clarkbsdague: yeah in this particular case I think auto restarting is fine17:17
clarkbfungi: it might be17:17
*** moted has quit IRC17:18
*** markmcclain has joined #openstack-infra17:18
*** matty_dubs|lunch is now known as matty_dubs17:18
*** moted has joined #openstack-infra17:20
*** basha has joined #openstack-infra17:20
*** sarob_ has joined #openstack-infra17:21
*** sarob has quit IRC17:21
*** kiall_ has joined #openstack-infra17:21
*** datsun180b_ has joined #openstack-infra17:22
*** dafter has quit IRC17:23
*** datsun180b has quit IRC17:24
*** datsun180b_ is now known as datsun180b17:24
clarkbmordred: looking at mysql permissions. I can grant CREATE to openstack_citest at a global level allowing it to create databases, is there a sane way to grant DROP to only the tables created by openstack_citest?17:24
*** tvb|afk has joined #openstack-infra17:25
*** tvb|afk has quit IRC17:27
*** tvb|afk has joined #openstack-infra17:27
*** tvb|afk has quit IRC17:27
*** tvb|afk has joined #openstack-infra17:27
openstackgerritSean Dague proposed a change to openstack-infra/config: auto restart elastic recheck on update
*** dizquierdo has left #openstack-infra17:29
*** DennyZhang has quit IRC17:30
*** dcramer_ has quit IRC17:32
*** basha_ has joined #openstack-infra17:32
*** beagles has quit IRC17:32
*** basha has quit IRC17:32
*** basha_ is now known as basha17:32
sdagueclarkb: the notify is all we need, right?17:33
*** dcramer has joined #openstack-infra17:33
*** sarob_ has quit IRC17:33
*** sarob has joined #openstack-infra17:34
clarkbyes, we may need to double check that that won't restart the bot every 10 minutes, but that is all you need to trigger service restarts17:35
*** dwcramer has joined #openstack-infra17:36
*** dcramer has quit IRC17:36
*** vogxn has quit IRC17:36
*** dwcramer has quit IRC17:37
*** dcramer_ has joined #openstack-infra17:37
*** tvb|afk has quit IRC17:37
hogepodgeping pabelanger17:37
*** beagles has joined #openstack-infra17:38
*** dafter has joined #openstack-infra17:38
*** dafter has quit IRC17:38
*** dafter has joined #openstack-infra17:38
*** sarob has quit IRC17:38
*** tvb|afk has joined #openstack-infra17:40
*** tvb|afk has quit IRC17:40
*** tvb|afk has joined #openstack-infra17:40
pleia2clarkb: that exec should only happen when the vcsrepo changes (unless it's sourced elsewhere)17:40
*** marktraceur has joined #openstack-infra17:41
hub_capmordred: <-- cli refactoring is done, im trying to get people to +2 it now (+6k line diff :/)17:42
*** yolanda has joined #openstack-infra17:42
clarkbpleia2: yeah, but vcsrepo I think isn't smart enough to only do it when the repo changes, iirc it may trigger subscribers/notifications whenever it runs17:42
*** dafter has quit IRC17:42
*** thomasm has quit IRC17:43
*** bnemec_ is now known as bnemec17:44
pleia2clarkb: boo, bad vcsrepo17:44
*** marun has joined #openstack-infra17:44
*** mdenny has quit IRC17:44
*** mdenny has joined #openstack-infra17:45
pabelangerhogepodge: pong17:45
clarkbpleia2: there are ways around that like have puppet and python fight over a file when actually does work so that a file subscription will notice (warning hack)17:45
pleia2yeah, it really shouldn't work this way though17:45
clarkbya, it is a bug17:46
*** hogepodge has quit IRC17:46
clarkbI think etherpad scales relatively well if you let apache open connections for it, load does jump when you point a couple hundred users at it but it falls off quickly once those processes fork and connections establish17:47
clarkbI will write a followup puppet change that tunes apache a bit17:47
*** marun has quit IRC17:47
clarkbfirefox really doesn't like the pads that are created by the stress tester though, ETOOMUCHDATA I guess17:47
*** marun has joined #openstack-infra17:48
*** basha_ has joined #openstack-infra17:48
*** oubiwann_ has joined #openstack-infra17:48
*** basha has quit IRC17:49
*** basha_ is now known as basha17:49
*** hogepodge has joined #openstack-infra17:49
*** pabelanger has quit IRC17:52
sdagueclarkb: they > 800 lines?17:53
*** hogepodge has quit IRC17:53
sdaguethat's where I've noticed slowdowns17:53
clarkbsdague: no the stress tester dumps a ton of data on one line17:56
clarkbline length may be part of the problem17:56
*** sandywalsh has left #openstack-infra17:57
*** boris-42 has joined #openstack-infra17:58
*** blamar has quit IRC17:58
sdagueoh, yeh that could be17:58
sdaguechrome is also better than ff with etherpad17:59
clarkbsdague: really? I have the opposite experience :)17:59
sdaguewell, at least at the rev I have :)17:59
mtreinishsdague: really, I've never had any extra issues with ff and etherpad18:01
clarkbsdague: could be versons of firefox and chrome too. They both move so quickly now18:01
*** fbo_away is now known as fbo18:01
*** hogepodge has joined #openstack-infra18:02
*** thomasm has joined #openstack-infra18:03
hogepodgemgagne: You had approved this one before, but there was a change to it. Out of order on the release.18:03
hogepodgegah, wrong room18:04
hogepodgesorry about that18:04
*** blamar has joined #openstack-infra18:04
*** thomasm has quit IRC18:04
*** mrodden has joined #openstack-infra18:04
*** thomasm has joined #openstack-infra18:05
openstackgerritClark Boylan proposed a change to openstack-infra/config: Tune apache for etherpad lite.
clarkbpeople that know more about apache. Is ^ insane?18:07
pleia2clarkb: how much ram on the etherpad server?18:07
clarkbpleia2: 2GB18:07
*** _david_ has joined #openstack-infra18:08
*** blamar has quit IRC18:08
fungiclarkb: if a lot of the load is forking apache workers, tune apache's minimum idle worker level upwards to buffer the impact?18:09
clarkbfungi: oh good idea let me test that change really quick18:09
_david_mordred, jeblair, clarkb FYI: there is currently discussion on repo-discuss ML about WIP workflow:!topic/repo-discuss/6V769w5zQok18:10
fungithere are several knobs around the worker pool you can use to determine available workers, max workers, kept/reclaimed workers, worker turnover, et cetera18:10
fungibeen a while since i played with them, but if memory serves that's been mostly the same since 2.018:11
_david_WIP Gerrit plugin is ready, core change to make it work (more or less 3 lines) was rejected by mfick. So it is included in the wip plugin itself18:11
clarkbfungi: I am just going to go for simple start an extra worker and set minimum Spare threads to greater than it was before and see if that helps18:11
clarkb_david_: the core change was to support the plugin ACL use?18:12
_david_clarkb, Nope18:12
_david_clarkb, zaro's "Change Owners group" change is still pending for review and is going to be merged18:13
*** DinaBelova has joined #openstack-infra18:13
_david_wip-plugin is Shrews's change migrated to Gerrit-2.8 and extracted as a plugin18:14
_david_so basically 3 lines needed to be added to Gerrit core to make it work: extend Change.STATUS enum with new WIP status + filter dash boards18:14
clarkb_david_: gotcha18:15
*** mrmartin has joined #openstack-infra18:15
_david_Unless some of you want to answer to dougk, i would just quote mordred answer to sop question on Shrews's change?18:16
*** emagana has joined #openstack-infra18:17
clarkb_david_: do you have a link to what mordred say back then? (eg to the change)18:17
_david_clarkb, i have everything  ;-)18:18
_david_WIP-Plugin: , Shrews's change:
clarkboh its there in your mail I see it, thanks18:19
clarkb_david_: I think the first four paragraphs of what mordred said are relevant. I am curious to see if mordred has anything to add since then (we have been using the feature for a lot longer now)18:22
lifelessttx: yeah because blocking every other transaction in the system for 20 seconds is so much better...18:22
_david_Reading it again: I think i should repost the sop's question too? Otherwise mordred's answer is may be out of context?18:22
_david_clarkb, that's the reason why i am here to ask you guys, if you have something to add to that answer made 1 year ago?18:23
clarkb_david_: mordred's comment doesn't address the "How is it different than a draft?" question though. WIP isn't intended on being private, instead it is a state that clearly indicates this change is not ready for review. Drafts are broken because they try to be private and instead aren't and until recently didn't emit events on state changes and so on18:25
_david_clarkb, year, you are right18:26
clarkbThe big difference between WIP and draft is that there is no implied security of a thing that is WIPed. Instead it exists to give reviewers the info they need to make better reviewing decisions18:26
_david_clarkb, you are invited to answer to dougk?18:27
fungiyou might say to someone "hey, here's this thing i'm working on, but it's in no way ready to seriously consider yet"18:27
_david_with quoting mordred?18:27
_david_fungi, but then he would answer: cool, then push it as a draft and add him as a reviewer18:28
fungibut i think more to the point that we use wip changes to allow us to keep our change development state stashed in gerrit so we risk less if we lose our local working/topic branches18:28
clarkbfungi: simply bumping the spare thread numbers seems to have helped. I will update my change18:29
openstackgerritClark Boylan proposed a change to openstack-infra/config: Tune apache for etherpad lite.
clarkbfungi: ^18:29
*** DinaBelova has quit IRC18:29
*** rnirmal has joined #openstack-infra18:30
fungithere's so much broken with drafts though, particularly when it comes to the event stream,interdependent changes some of which are drafts while others aren't, patchsets in a single change where some are drafts and others aren't, et cetera. not to mention the whole "people think setting something as a draft secures it against third-party access"18:30
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Fix test_files_at_url_pass
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests
clarkbfungi: they have fixed some of thosethings recently18:30
clarkbfungi: but you are right it is particularly jarring when some patchsets are draft and others aren't and so on18:31
clarkbI don't think you can completely fix those problems18:31
fungii think it's mainly that drafts don't fit into our established workflow as a community, so it would be nice to replace them with something which is actually public or at least be able to disable the drafts feature entirely18:31
fungioh, another gripe with drafts, you can't set an already-public patchset into a draft state, which is another way it differs from what we need out of work-in-progress18:32
_david_yeah, these features complete each other, very cool with drafts: you can remove them entirely from the system18:33
fungiwhich, again, i think goes back to drafts trying to be some sort of security feature and the idea of already-published data being made secret doesn't fit with their threat model18:33
fungiexcept gerrit has so many other places which leak data (git replication, gitweb, change number guessing with anonymous access enabled, and so on) that actually securing drafts is a tall order18:35
*** gyee has quit IRC18:35
clarkbfungi: any idea why etherpad-dev doesn't show up on cacti? I am in /var/lib/cacti/rra and there are etherpad-dev files18:36
fungimmm, looking18:36
*** pblaho has joined #openstack-infra18:38
*** fbo is now known as fbo_away18:38
*** DinaBelova has joined #openstack-infra18:41
pleia2clarkb: speaking of which, my cacti change for adding nodepool to cacti went in last month (47614) but it never ended up showing up, I recall you said it needed a nudge server-side?18:41
clarkbpleia2: it might need snmpd kicked, however in the past that was necessary when blank graphs were rendered18:42
clarkbpleia2: in this case no graphs are showing up at all18:42
pleia2clarkb: yeah18:42
clarkbpleia2: which is similar to etherpad-dev. could be something else going on18:42
* pleia2 nods18:42
jog0dumb question: when bug reference is added as an amendment, the launchpad is not updated automatically with a link to the change that fixes the bug. Is it not the case?18:42
clarkbjog0: right bug needs to be in the first patchset18:43
jog0clarkb: WAAA18:43
jog0that is not cool18:43
clarkbjog0: this was done in an effort to be stateless and not spam bugs18:43
jog0clarkb: it has bad side effects18:43
fungiclarkb: i see etherpad-dev in the host table in cacti's mysql db18:43
*** hashar has joined #openstack-infra18:43
clarkbfungi: what about nodepool?18:43
pleia2jog0: it will link to the bug though, so when it's merged it'll update the bug18:43
jog0bugs don't get updated properly, and people are creating duplicate patches18:44
pleia2(I made the same mistake recently)18:44
fungiclarkb: pleia2: yep, they're leaf id 45 and 46 i think18:44
pleia2just not while under review18:44
fungibut not being rendered18:44
jog0pleia2: oh thats good, as long as it updates when it gets merged18:44
clarkbfungi: 45 and 46 are the git servers according to the web ui18:44
fungiclarkb: are you sure? i show those as 40-4318:45
fungiand sorry, nodepool and etherpad-dev as 44 and 45 not 45 and 5618:45
fungiso they are. the id column in the host table may not match the leaf id in the web interface after all18:46
clarkb44 is git04, 45 is git03 on the webui18:46
fungiyep, that's what i see in the browser too18:47
fungiyeah, spot-checking the list if host ids in the table, none of it matches up and it's not a constant offset either18:48
fungiso that leaf id number likely comes from somewhere else18:49
*** DinaBelova has quit IRC18:50
fungialso there can't be a 1:1 correspondence anyway sinc there are multiple trees18:51
clarkboh interesting I cna hit the graphs directly, I just don't see the server in tree mode18:52
clarkb for example18:52
clarkb is a fun one18:53
fungii'm trying to figure out if there's somewhere it extracts the db contents out to a file or something which is stale, because i'm not seeing anything obvious missing for them in the db18:53
*** alcabrera|afk has quit IRC18:54
clarkbpleia2: same thing with nodepool data is being collected and rendered, we just don't get the nodes in thte tree view18:55
*** pabelanger has joined #openstack-infra18:55
pleia2well, that's something18:55
mordredclarkb, _david_: I have made a response to the thread on the mailing list18:56
mordred_david_: where is your patch to core gerrit that got rejected?18:56
mordred_david_: and how did you include it in your plugin?18:56
annegentle_hey anyone an eavesdrop expert? I'm looking for logs from the TC meeting where an API lead was proposed and rejected... and I only see 10/12 and forward on eavesdrop18:59
annegentle_anyone have an idear?19:00
annegentle_ah! I have an idear. Look before we had labeled meeting folders19:00
_david_mordred, thanks! i appreciate you help19:01
mordred_david_: is this the one that got blocked: ?19:01
_david_mordred, exactly19:02
mordred_david_: would you be willing to restore it?19:02
_david_mordred, sop was going to approve that19:02
mordred_david_: I am going to beg them to reconsider19:02
_david_mordred, sure, i would19:02
mordred_david_: I mean, for the love of god, one of the biggest gerrit installs in the world is begging you to land a tiny patch19:02
mordredthat is in support of a much larger patch that you already rejected that the work went in to pulling out as a plugin19:03
mordredI think "this should be a generic workflow engine" is completely unreasonable19:03
_david_mordred, you told me that? I m totally pissed off, and shut down every contribution on Gerrit-Review19:03
_david_mordred, exactly19:03
*** datsun180b has quit IRC19:03
mordredso, if you'll restore that change, I'll go email mfick and sop19:04
*** datsun180b has joined #openstack-infra19:04
_david_that the biggest boolshit that i've heard from mfick till now19:04
*** pabelanger has quit IRC19:04
_david_mordred, done19:04
_david_mordred, Please help me here ...19:04
mordred_david_: thank!19:05
_david_mordred, Thank you!19:05
*** pblaho has quit IRC19:08
annegentle_clarkb: winner winner! thanks19:08
_david_mordred, have you checked the screen shots?
mordred_david_: yes! quite awesome!19:10
Shrews_david_: that's pretty cool19:10
*** vipul is now known as vipul-away19:10
*** vipul-away is now known as vipul19:10
fungimordred: scroll up a bit in #gerrit and revel in the surprise amongst other users that the verify label is being removed. because clearly you should only have reviewers no verifiers19:10
_david_mordred, yeah!19:11
mordredI have just started begging in two different places19:11
mordredthey removed verify?19:11
_david_Shrews, it's your Baby !!! and it's your code19:11
fungimordred: my thoughts as well :/19:11
_david_mordred, yeah, they did19:11
Shrewstime to fork gerrit19:11
clarkbmordred: but they made review categories configurable through git19:12
_david_mordred, we are talking about three line of code in core .... Who care???19:12
clarkbso removing verified isn't a big deal19:12
clarkbyou just add it to All-Projects.git and move on19:12
_david_mordred, clarkb 3 lines of code in All-Projects or per project base19:12
_david_mordred, i am going to help with upgrading openstack Gerrit installation & configuration19:12
fungioh, so you can have some projects with a verify column and others without. interesting19:13
_david_exactly + other labels per project base19:13
mordred_david_: thank you for your help on this - it's super exciting19:13
*** dmakogon_ has joined #openstack-infra19:13
_david_mordred, you know, you guys helped /me us first ;-)19:13
clarkbfungi: right, we could move some of our non voting jobs to special review categories19:13
clarkbfungi: and have columns for License Check in requirements and so on19:13
mordredclarkb: ++19:14
_david_mordred, clarkb i tried to convince mfick & sop here:19:16
_david_David Purshouse: Not sure about work in progress though.   Both Martin and Shawn have expressed doubts about it, so maybe it's better to defer to a later release.  Unless you are able to convince them otherwise...19:17
* _david_ I think i've convinced Shawn and am very close to get his +1 on the latest ps of that change[1].19:17
_david_With these 10 LoC in core we could let 1000 LoC WIP workflow in wip-plugin[2].19:17
_david_check what sop answered ... can you see it?19:17
*** blamar has joined #openstack-infra19:17
clarkbI need to grab lunch, back in a bit19:18
openstackgerritDavid Kranz proposed a change to openstack-infra/devstack-gate: Don't check logs for ERRORs with neutron
mordred_david_: hopefully I can make some headway somewhere19:19
mordredwith the begging19:20
_david_mordred, Wau!19:20
_david_mordred, just have read it...19:20
*** CaptTofu has quit IRC19:20
*** CaptTofu has joined #openstack-infra19:21
mordred_david_: :)19:21
_david_mordred, just to mention: /me wrote very private mail to sop begging to help me to help OpenStack project!19:21
_david_three lines of code: Who care???19:21
*** pabelanger has joined #openstack-infra19:21
*** DennyZhang has joined #openstack-infra19:22
mordredthat's what I'm saying19:22
*** dmakogon_ has left #openstack-infra19:22
*** vipul is now known as vipul-away19:25
*** vipul-away is now known as vipul19:26
*** dprince has quit IRC19:27
fungiclarkb: pleia2: interestingly cacti indicates its awareness of etherpad-dev and nodepool in list view, just not in tree view19:29
mordredclarkb: I'm curious about having specific labels for some of the tests19:33
mordredclarkb: and extending zuul with label awareness19:33
mordredclarkb: and potentially not needing to run _everything_ on a recheck19:34
mordredclarkb: like if you could recheck a particular label's tests19:34
mordredso "style, unit, integration" for instance19:34
fungiclarkb: i'm starting to wonder whether the addition of the git farm tree broke our ability to add new hosts the way we had been19:34
fungiclarkb: i think maybe we have to indicate the host tree now that it's no longer the only tree19:35
*** yolanda has quit IRC19:36
*** sdake has joined #openstack-infra19:37
*** hashar has quit IRC19:40
*** mrmartin has quit IRC19:40
fungiclarkb: pleia2: have them on there now... apparently needed to:19:41
fungicd /usr/share/cacti/cli && sudo php -q add_tree.php --type=node --node-type=host --tree-id=1 --host-id=4419:41
pleia2oh, obviously ;)19:41
pleia2thanks fungi19:41
fungiwhere the tree id is determined by running add_tree.php --list-trees and the host id by running add_tree.php --list-hosts19:42
funginot sure what the best way to puppet this is going to be19:42
pleia2yeah, hm19:42
mordredRyan_Lane: if wikimedia wanted to throw some support towards too ...19:43
fungiprobably yet-another-shell-script19:43
mordredsdague: perhaps IBM too ?19:43
sdaguemordred: if you think it would help19:43
fungiclarkb: pleia2: i think i'll just open a lhf bug with the appropriate details for the moment, and then if someone else fixes it first i won't complain19:44
*** hashar has joined #openstack-infra19:44
pleia2fungi: wfm19:44
mordredsdague: I have no idea19:44
Ryan_Lanemordred: let me point ^demon at it19:45
mordredRyan_Lane: thanks! I think I've reached the point of begging borrowing and stealing on this one19:46
_david_Ryan_Lane, please pock  ^demon19:47
Ryan_Lanewe -1 our own changes right now or write "WORK IN PROGRESS" in the description, but that sucks19:47
_david_Ryan_Lane, please if Chad could beg for it???19:47
_david_Chad is considered as very important Gerrit community member19:48
fungidon't spend him all in one place ;)19:48
Ryan_LaneI wouldn't mind if this just replaced drafts, personally. I think drafts is a worthless feature19:48
mordreddrafts is completely worthless19:48
_david_[...] and sop hear what wikimedia guys are saying ;-)19:49
Mithrandirisn't the problem with drafts they work in unexpected ways?19:49
mordredI can think of at least 219:49
Ryan_Lanehm. actually, the person that -2'd has a pretty good point19:49
Ryan_Lanegeneric statuses would be nicer19:49
_david_DRAFT have only one really nice feature: they can be deleted19:49
mordredone: if you upload a draft change to a change that has a non-draft patchset, you see the old patchsets but can no longer vote19:49
mordredtwo: they replicate, so are not private for real19:50
mordredRyan_Lane: they would be nicer - but this is a pretty small patch, would make the current user base happy, and would give some space to get to a generic status feature19:50
mordredin fact, if I could get us on to 2.8, I might be able to justify having someone work on generic statuses19:51
_david_mordred, that's the point! in 3.0?19:51
*** hashar has quit IRC19:51
mordredalso, this patch is the THIRD fully operational attempt we've submitted to them in an attempt to express the thing we need19:51
Ryan_Lanemordred: well, can't you guys manually merge in the feature for now?19:51
mordredRyan_Lane: we dont' want to carry a new fork19:52
* Ryan_Lane nods19:52
_david_Guys, may be you missed that, but i spent another couple of days, turning WIP plugin to the core feature again (like it was with Shrews's original change):19:52
mordredwe'd like to start running fairly CD off of their releases19:52
mordredyup. ok. so this is attempt 4 then19:52
mordredI mean, work with us here19:52
_david_Ryan_Lane, with this there are no any arguments against iit:19:52
_david_Change.STATUS.WIP is used for core Gerrit feature, like DRAFT, but noch sucked19:53
Ryan_Lanemordred: yeah, I've heard it can be kind of a pain to work with gerrit19:53
Ryan_Lanethey make odd design decisions19:53
*** hashar has joined #openstack-infra19:54
_david_Ryan_Lane, can you poke Chad on it?19:54
Ryan_Lanealready did19:54
mordredthey do - almost every time we've gone to them with something, there has been weird and crazy pushback19:54
mordredRyan_Lane: you are awesome19:54
_david_Ryan_Lane, thanks, appreciate it19:54
Ryan_Lanehe's in #wikimedia-operations, if you'd like to talk to him directly19:54
_david_let me beg him19:54
Ryan_LaneI couldn't really gauge his reaction ;)19:54
Ryan_Laneit seemed like "yep, that's a feature alright" :D19:55
lifelessmordred: CD + releases - lost me :)19:55
ttxlifeless: as long it's mine that runs I don't mind :P19:55
mordredlifeless: don't think about it too much :)19:55
lifelessttx: sadly thats how LP ended up slow for /everyone/19:56
lifelessttx: and design and code defects like the one that affects that critical got /into/ the codebase - because noone could actually see the damage being done.19:57
mordredttx: that's ok - maybe at some point we won't have the issue anymore19:57
ttxmordred: heh19:57
*** hashar has quit IRC19:58
*** sdake_ has quit IRC19:58
sdaguemordred: I wonder if we know anyone in the denver area that could take martin out for beers and explain?19:58
*** hashar has joined #openstack-infra19:58
mordredsdague: is that where he is?19:58
mordredI could fly to denver19:58
mordredjeblair likes skiing ...19:58
*** sdake has quit IRC19:58
fungisend dkehn to entertain him19:59
sdagueI was just trying to figure out if I knew folks that knew martin well enough to guage his responses19:59
fungior is he in boulder19:59
*** sdake has joined #openstack-infra20:00
*** sdake has joined #openstack-infra20:00
openstackgerritMatthew Treinish proposed a change to openstack-infra/elastic-recheck: Convert test_classifier to be a unit test
mordredclarkb: got a sec? (or fungi)20:00
mordredlifeless: that's where it switches to using pypi-mirror20:01
lifelessI see a different venv20:02
lifelessmordred: but it still calls run-mirror20:02
dkehnfungi, yep in boulder20:02
mordredlifeless: yes20:02
mordredlifeless: run-mirror is now provided by pypi-mirror and not by jeepyb20:02
*** mdenny has quit IRC20:02
lifelessoh, its factored out of jeepyb?20:03
mordredit's a new project now20:04
lifelessSpamapS: ^20:04
SlickNikhey guys.20:04
mordredSpamapS: I'm sorry, did I not mention that?20:04
mordredlook. it's SlickNik20:04
lifelesswe just added jeepyb based elements for this you see...20:04
*** mdenny has joined #openstack-infra20:04
*** sdake has quit IRC20:05
SlickNikhad a quick question to do with LaunchPad permissions for openstack projects.20:05
fungimordred: on 48355 i'm a little confused by the encode vs decode operations there. you're indiscriminately decoding subprocess output from utf-8 (to unicode) but then reencoding it to return utf-8 from _get_git_directory() if running py2.x and presumably returning straight unicode otherwise?20:05
mordreddhellmann: ^^ ?20:05
SlickNikSeems like contributors should be able to edit / add information to blueprints, right?20:06
SlickNikFor trove for some reason, contributors can't edit all fields in a blueprint.20:06
*** pentameter has quit IRC20:06
SlickNikAny idea how I go about debugging/fixing this?20:06
SlickNik(I don't know much about LaunchPad permissions, so apologies in advance if this seems like a silly question.)20:07
clarkbmordred: dhellmann: re using the u'' makes python3 < python3.3 not possible20:08
clarkbmordred: dhellmann: would returning '' work?20:09
*** dcramer_ has quit IRC20:09
SpamapSmordred: Thats good to know I'll fix up my aptly named "pypi-mirror" element :)20:09
fungi_run_shell_command() already returns '' instead of u'' when there's an empty response20:09
fungibut calls .decode('utf-8') when not20:09
*** sdake has joined #openstack-infra20:09
clarkbfungi: get_version_from_git() returns u'' in the patch20:09
fungiclarkb: yep, i was pointing out the inconsistency20:10
mordredSlickNik: not all contributors can edit all fields in blueprints - some are for the ... ttx, what's the team permission for that?20:10
fungimordred: dhellmann: also has this been tried under locales != utf-8? does subprocess always return utf-8? (though that's being decoded in existing code not touched by this patch anyway)20:11
SlickNikAh, thanks mordred! Is there a guideline as to which fields contributors should be able to edit?20:11
SlickNikOkay, took a look at a couple of projects and it seems like it's a standard ("series goal", "whiteboard", and "work items")20:13
*** _david_ has left #openstack-infra20:13
mordredSlickNik: I believe it's the Driver team that can edit those20:13
mordredthe others20:13
*** DennyZhang has quit IRC20:13
mordredfungi, clarkb: thanks for the feedback - hopefully dhellmann will bring the sanity-based answers20:14
SlickNikGot it. That clears things up. Thanks mordred!20:14
clarkbfungi: I don't think other locales would be supported20:14
*** pentameter has joined #openstack-infra20:14
clarkbor rather non utf8 locales20:14
fungiprobably needs to say so in big letters on the outer wrapper if that's the case20:15
clarkbI didn't think it was a problem but it looks like git finally grew support for unicode so it may matter20:15
dhellmannmordred, fungi : reading backlog...20:15
*** CaptTofu has quit IRC20:16
mordredyay it's dhellmann !20:16
*** CaptTofu has joined #openstack-infra20:16
dhellmannfungi: python 2 doesn't want unicode strings as args to subprocess, so that's why the git directory needs to be encoded20:16
dhellmannclarkb: good point, that u'' should be a six text type thing20:17
dhellmannfungi: yeah, that should probably use the system default encoding20:17
SpamapSmordred: so should I pip install pypi-mirror rather than git cloning it?20:18
SpamapSah no, no releases20:18
dhellmannfungi: the u'' change was a late addition, so I should go back and make _run_shell_command() return unicode, too20:18
fungidhellmann: ahh, got it. so the actual intent is that _get_git_directory() should return utf-8 under python 2.x and return raw unicode under 3.x?20:19
dhellmannso yeah, this patch isn't super consistent :-/20:19
dhellmannI suppose it would make more sense to re-encode the git dir where it is being used as a command line argument20:20
*** oubiwann_ has quit IRC20:20
fungii would have probably been less confused if the encoding decision was made where the function was called rather than within the function, but it makes more sense now that i have context20:20
*** oubiwann_ has joined #openstack-infra20:20
fungiright, what you just said20:20
* mordred sort of thinks all of pbr could probably do with a unicode/utf-8 overhaul20:20
fungithe decision is being made too far from where its impact is felt20:20
* dhellmann thinks mordred is right20:20
dhellmannfungi: yeah, I'll update that20:21
*** SergeyLukjanov has quit IRC20:21
dhellmannfungi: you know, I was probably trying to be consistent with what I did in _get_version_from_pkg_info(), since I don't have access to the user of the data for that function20:22
fungiyah, makes sense there20:23
dhellmannfungi: and for the utf-8 question, I think I copy-pasta-ed from other similar calls that were already in the file :-/20:23
*** SergeyLukjanov has joined #openstack-infra20:23
fungidhellmann: totally, it was more of a general "does any of this work on systems with non-utf-8 locales?" (not just your patch)20:24
dhellmannfungi: so it's ok to address that separately?20:24
fungii suspect that, as mordred points out about unicode handling overhaul, there is probably some system locale detection missing in pbr overall20:25
fungiand assumptions about everyone using utf-820:25
mordredI believe its utf-8 awareness has grown a bit like mold20:25
*** SergeyLukjanov has quit IRC20:25
fungiwell, we all use utf-8 because we're smart and we know there's no future in legacy encodings. which is why we don't notice if non-utf-8 systems break as a result ;)20:26
dhellmannfungi: there are several users of _get_git_directory(), so that may have contributed to the "fix it in one place" approach as well20:26
*** anteaya has joined #openstack-infra20:26
fungidhellmann: makes sense then, probably just needs a slightly more expanded comment line that the lone url20:27
dhellmannfungi: I'll do that20:27
fungier, than that lone url20:27
fungisomething suggesting that consumers of that function may expect different data types depending on the python version20:27
dhellmannhmm, that url doesn't even seem relevant20:27
clarkbfungi: mordred how does that look for an etherpad upgrade notice?20:28
lifelesspleia2: so, is that experimental queue available now ?20:29
fungilifeless: if you're asking what i think you're asking, search in layout.yaml for the word experimental to see examples20:30
fungiit's very much in use20:30
lifelessfungi: I'm not asking what you think I'm asking20:30
mordredfungi: he's asking about a specific experimental job20:30
fungilifeless: then please ignore ;)20:30
pleia2lifeless: no, jeblair has some work to do on nodepool before changes in the merged queue can be pushed in20:30
pleia2lifeless: currently puppet is turned off while it's being resolved, so my change isn't applied yet20:30
mordredclarkb: it does not look like it has content20:31
* lifeless wants to be able to get that out of the working-set20:31
clarkbmordred: oh I may still be talking to the new server :/ /me checks /etc/hosts20:31
*** dcramer_ has joined #openstack-infra20:31
fungii was about to suggest the same20:31
mordredclarkb: what's the new ip? I can just check it out20:31
fungii strongly suspect you are20:32
* mordred curious to see our new etherpad overlord20:32
clarkb166.78.29.95 is the new IP, I will move this etherpad to the current server20:33
*** pabelanger has quit IRC20:33
mordredclarkb: you need to shift-reload old etherpads btw20:34
clarkbmordred: did that fix the cache thing for you?20:34
*** hogepodge_ has joined #openstack-infra20:34
*** hogepodge has quit IRC20:34
*** hogepodge_ is now known as hogepodge20:34
fungiclarkb: might also mention you've load-tested the configuration to better accommodate the summit activity volume20:34
mordredclarkb: well, without shift reload, js pages it was using were wrong20:34
clarkbmordred: yeah, I had to actually clear my caches20:34
mordredclarkb: you also may want to add "databases running on trove"20:35
clarkbfungi: but then people might get the impression I know what I am doing :P20:35
clarkbmordred: fungi both good ideas, adding that content now20:35
dhellmannmordred, fungi : I am unable to reproduce the unicode issue that prompted me to change the git dir return type :-/20:39
clarkbspeaking of identifying and correcting scale issues. if doesn't look completely insane I will apply it to new etherpad and new etherpad-dev20:39
mordreddhellmann: yah!20:42
*** cody-somerville has quit IRC20:42
anteayaclarkb: etherpad looks good to me20:43
mordredsoren: I was wrong. # is terrible20:44
mordredsoren: I apologize for being a moron20:44
clarkbanteaya: thanks20:44
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add unit tests to gating jobs on elastic-recheck
*** matty_dubs is now known as matty_dubs|gone20:44
*** boris-42 has quit IRC20:45
*** boris-42 has joined #openstack-infra20:47
sdagueclarkb: so maxclients is really high in that -
sdagueI don't typically use mpm_worker, any idea what the per thread memory add is?20:48
clarkbsdague: no hard numbers, but its seems relatively low, I haven't been able to get above ~400MB when throwing a couple hundred etherpad users at it20:49
dhellmannclarkb: it looks like pbr doesn't already use six. have any suggestions for fixing that u'' issue in a portable way without adding a dep?20:49
*** sarob has joined #openstack-infra20:49
sdagueclarkb: were you anywhere near max clients then?20:49
dstufftbundle six20:49
dstufftsix is a single file20:49
dstufftit's easy to bundle20:50
clarkbsdague: it looks like there is a >1 connection per etherpad user (I think http connection +
dstufftit's BSD licensed as well20:50
* dhellmann gives dstufft a look20:50
clarkbsdague: when I had it at max clients for 1024 I got close ~90020:50
*** CaptTofu has quit IRC20:50
dstufftdhellmann: you're writing packaging tools, get used to bundling20:50
clarkbsdague: so roughly 1GB for 2048 clients is what I guestimated20:50
*** CaptTofu has joined #openstack-infra20:50
jeblairmordred, clarkb, fungi: hi, i'm back20:50
dstufftpip is up to like 4 or 5 things bundled now20:51
clarkbsdague: but I defer to your knowledge, I am definitely not well versed on apache20:51
*** sarob has quit IRC20:51
sdagueclarkb: yeh, as long as you think you won't drive into swap, I'd say it's sane20:51
sdagueall my stuff typically needs prefork20:51
sdagueso I don't know worker's quirks20:51
*** sarob has joined #openstack-infra20:51
clarkblet me start up a longer running stress test so that we can get decent numbers in cacti20:52
*** sandywalsh has joined #openstack-infra20:52
mordredjeblair: o hai!20:54
*** sarob_ has joined #openstack-infra20:54
mordredclarkb: why do we care about python3 < 3.3 ?20:54
mordredclarkb: we have zero historical python3 users20:54
mordredany new python3 users should _clearly_ be on >=3.320:55
*** sarob has quit IRC20:55
*** sandywalsh has quit IRC20:55
jeblairi'm working on nodepool now20:55
hub_capmeeting-ext #topic is not changing20:57
*** sandywalsh has joined #openstack-infra20:57
clarkbmordred: that isn't true20:57
clarkbmordred: wheezy and precise are both python3.220:57
clarkb(I am not concerned about server projects, but this is an issuefor clients and oslo and so on)20:57
*** anteaya has quit IRC20:58
*** hogepodge has quit IRC20:59
*** ArxCruz has quit IRC21:00
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: Do not pass unicode where byte strings are wanted
dhellmannfungi, mordred, clarkb : removed the unicode handling of git dir, left the version stuff, and tried to be more portable for other versions of python 3 in ^^21:02
fungihub_cap: the bot was -o in both meeting channels. fixed now21:02
fungiclarkb: on 50700, any sane way to restart apache when /etc/apache2/conf.d/connection-tuning changes?21:03
clarkbfungi: not really, as many of those values require a hard restart21:03
clarkbfungi: I figured it was easiest to just do it by hand when it wouldn't hurt21:03
fungii take it the apache module does a sighup reload21:04
jeblairclarkb: i think it might be okay to restart apache on etherpad at any time21:04
sdagueclarkb: so on the recheck bot notify, you want me to create a work around for that where I use some kind of state file, or do you think vcsrepo won't restart us every 10 mins21:06
fungijust thinking forward to when we're in hk and the etherpad server is falling over because we misjudged the tuning21:06
clarkbsdague: I am not sure, we can try it as is and see if it gets noisy21:06
sdagueok, if we could, that would be cool21:06
sdaguethen I can build the work around if now21:06
sdagueif not21:06
clarkbfungi: good point, I am looking at the puppetlabs apache stuff now to see what it does21:06
jeblairnew nodepool is running:
fungijeblair: so we're safe to turn the puppet agent back on, or is that still on your dev box?21:07
dhellmannhey, I'm getting errors from uvirtbot in openstack-meeting. Anyone know what that is?21:08
jeblairfungi: i want to watch it for a few more mins to make sure it's not crazy (it's still burning through the buffer); soon i hope21:08
jeblairdhellmann: i think soren acted on mordred's request to change the commend character to '#' from '^'21:08
jeblair#isn't that right?21:09
uvirtbotjeblair: Error: "isn't" is not a valid command.21:09
*** melwitt has joined #openstack-infra21:09
*** shardy is now known as shardy_afk21:09
fungidhellmann: except we neglected to keep in mind that uvirtbot likes to complain when it thinks you're talking to it, even when you're not21:09
dhellmannand it is taking my commands with the #21:09
jeblairdhellmann: at any rate, it's independent from meetbot, so your commands should be working and you can try to ignore it21:10
dhellmannah, right, 2 bots21:10
dhellmannthanks, jeblair21:10
mordredfuck gerrit21:10
jeblairmordred: ?21:11
sorenmordred: Let's just change it again.21:11
fungisoren: is uvirtbot capable of being silent when it thinks it gets commands it doesn't recognize, or is that too complicated?21:11
mordredjeblair: I would like to officially fork it, maintain that fork, and be done with it21:11
mordredbut, I may be agitated21:11
mordredso please everyone ignore me for a moment21:11
fungiyou do _seem_ agitated21:11
sorenfungi: We can just tell it not to respond to anything unless explicitly addressed?21:11
jeblairmordred: oh, drat, i was about to rewrite it in python21:12
mordredjeblair: awsome21:12
sorenuvirtbot:  config supybot.reply.whenAddressedBy.chars21:12
uvirtbotsoren: Global: #; #openstack-infra: #21:12
*** pcm_ has quit IRC21:12
*** dkranz has quit IRC21:12
sorenUh.. No.21:12
sorenuvirtbot:  config supybot.reply.whenAddressedBy.chars --remove21:13
uvirtbotsoren: Error: Value must contain only ~!@#$%^&*()_-+=[{}]\|'";:,<.>/?21:13
sorenuvirtbot:  config --remove supybot.reply.whenAddressedBy.chars21:13
uvirtbotsoren: Error: 'supybot.--remove' is not a valid configuration variable.21:13
fungisoren: sounds ideal21:13
openstackgerritClark Boylan proposed a change to openstack-infra/config: Tune apache for etherpad lite.
clarkbfungi: jeblair sdague ^ that better? I am going to put that on etherpad-dev shortly21:14
fungisoren: assuming it still picks up on the bug patterns, anyway21:14
sorenfungi: Yup.21:14
sorenuvirtbot:  config supybot.reply.whenAddressedBy.chars ' '21:15
uvirtbotsoren: Error: Value must contain only ~!@#$%^&*()_-+=[{}]\|'";:,<.>/?21:15
sorenOh, ffs.21:15
mroddenanyone have any idea why distutils can't seem to find pbr even though i have it in dist-packages on ubuntu?21:15
*** uvirtbot has quit IRC21:16
mroddenrunning into this when i do python develop on nova21:16
*** uvirtbot has joined #openstack-infra21:17
fungimrodden: that's an exciting paste21:17
hashar"""<jeblair> mordred: oh, drat, i was about to rewrite it (Gerrit) in python21:17
fungimrodden: you get no output whatsoever? ;)21:17
hasharjeblair: mordred: you guys are crazy :]  I like it.21:17
sorenuvirtbot: config supybot.reply.whenAddressedBy.chars21:17
* mrodden sighs21:18
uvirtbotsoren: Global:  ; #openstack-infra:21:18
mroddenits been one of those days...21:18
*** uvirtbot` has joined #openstack-infra21:18
sorenuvirtbot`: Who the crap are you?21:18
fungijeblair: technically wouldn't that be reverting it to python (the predecessor of gerrit *was* in python, at any rate)21:18
uvirtbot`soren: Error: "Who" is not a valid command.21:18
jeblairfungi: except without the baggage? :)21:19
openstackgerritClark Boylan proposed a change to openstack-infra/config: Tune apache for etherpad lite.
clarkbpatchset 3 had a dependency cycle. Hopefully 4 is better21:19
*** uvirtbot` has quit IRC21:19
*** uvirtbot has quit IRC21:19
sorenComputers are difficult.21:20
*** uvirtbot has joined #openstack-infra21:20
*** dcramer_ has quit IRC21:20
*** hogepodge has joined #openstack-infra21:20
jeblairnew nodepool algorithm actually spinning nodes up ^21:20
clarkbps4 seems to have worked21:20
*** hashar has quit IRC21:21
clarkbso that is applied to etherpad-dev, feel free to hit it with hammers21:23
mrodden any ideas?21:25
mroddenquestions/comments/snide remarks...21:26
clarkbmrodden: looking now21:26
mroddeni can python -c "import pbr"21:26
clarkbmrodden: what version of setuptools do you have?21:26
fungidhellmann: does that still solve the errors you were getting on os x?21:26
clarkband do you have more than one setuptools?21:26
fungidhellmann: i notice you got rid of the git-dir unicode reencoding in the latest patchset21:27
clarkbmrodden: I would install the latest setuptools and remove any old versions if you are not already in that state21:27
mroddeni don't suppose i can use pip to do that21:28
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Fix stats after min_ready change
clarkbmrodden: pip freeze may show it21:28
mroddenactually it doesn't...21:29
mroddenthats weird21:29
mroddeni have 1.021:29
clarkboh so that needs fixing too21:29
mroddenwhere does pip read the packages from?21:29
clarkbmrodden: /usr/local/lib/pythonXX and /usr/lib/pythonXX on debian/ubuntu21:29
*** CaptTofu has quit IRC21:29
clarkbmrodden: basically old pip is bad and old setuptools is bad. You want to make sure you have the latest of both21:30
clarkbor at least pip >= 1.4 and setuptools >=0.721:30
*** CaptTofu has joined #openstack-infra21:30
mroddenopenstack@locale-dev:/opt/stack/nova$ pip --version21:30
mroddenpip 1.4.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)21:30
clarkbcool with new pip there is a different command that lists installed packages, pip list maybe? that should show you setptools21:32
clarkbI forget why freeze doesn't and dstufft and I argued about it21:32
mroddenok good to know21:32
*** svarnau_ has quit IRC21:32
mroddeni just broke pip for the moment21:32
mroddenpurging my system of all setuptools21:32
*** svarnau has joined #openstack-infra21:33
mroddenok setuptools 1.1.6 installed...21:33
mordredjeblair, zaro: ok. I calmed down and talked to mfick more, there is a decent chat in #gerrit about what he's looking for - and it's not actually an entire new state engine21:34
clarkbcool now see if pbr and nova and oslo are happy21:34
mordredbasic themes seem to be:21:34
mordreda) why do you guys think that coming in with code before getting buy in on direction is a good idea?21:35
mroddenoh crap .... i think i just realized what the problem is21:35
mroddeni have a pbr egg directory in my /opt/stack/nova21:35
mordredb) maybe turn the state enum into something similar to the approvals thing21:35
clarkbmordred: because a) when we have tried to show up before code no one understands21:35
dstufftclarkb: because generally you don't want to pin your setuptools to a specific version :V21:35
mordredclarkb: indeed. but I think he's saying that's how their community works and other people seem to have figured it out21:35
clarkbmordred: at one point I was trying to fix drafts, but that went nowhere because I didn't have any code21:36
clarkbit is really hard to have technical design and discussions (that was my experience at least)21:36
dstufftand it doesn't pin pip itself because upgrading pip using pip without existing the process requires trickery like fork or something to get around the fact the old pip is already imported21:36
dhellmannfungi: yes, I could not reproduce the issues with the git dir21:36
mordredclarkb: see most recent 4 lines from him21:36
clarkbmordred: I am no longer in that channel21:36
dhellmannfungi: so I'm choosing to believe it was transient or my imagination, and we can fix it properly if it shows up again21:37
mordredmfick | is to write a hack up as a change, upload it with [RFC] in the summary21:37
mordredmfick | then people will see the idea and understand that it is just a proposal that may imply more work, but it gives the idea a place to be21:37
fungidhellmann: okay, in that case one comment inline but otherwise lgtm on the version reencoding21:37
mordredthat seems to be a process step for them that might be important to note for us21:37
* fungi wonders if they need an "rfc" state for changes in gerrit ;)21:38
clarkbmordred: that doesn't work21:38
clarkbmordred: see zaros recent change, what happens is you get your first -1 then no one else reads the RFC21:38
clarkbbecause well it has a -121:38
mroddenclarkb: nova still isn't happy :(21:39
clarkbmrodden: is it the same error?21:39
*** _david_ has joined #openstack-infra21:39
mroddensort of21:39
mroddenlooks a bit diff. but its the same problem looks like21:40
jeblairpleia2: ping21:41
*** mriedem has quit IRC21:41
* pleia2 has a look21:41
jeblairpleia2: um, the password it put in the config file does not look like a good password.21:41
jeblairpleia2: assuming it's supposed to be a halfway decent password, that is likely (at least part of) the problem21:42
_david_mordred, his suggestion is clear, but... i think it can not be done for upcoming 2.8 release any more, as stable-2.8 is going to be cut this ,omth21:42
openstackgerritA change was merged to openstack-infra/nodepool: Fix stats after min_ready change
pleia2jeblair: have time for a quick call in a few minutes to work through this?21:42
mordred_david_: yah21:42
jeblairpleia2: yeah, lemme go through the plumbing real quick and see if i spot an error21:43
_david_and he hadn't any arguments against: why DRAFT is in tree ;-)21:43
pleia2jeblair: ok, thanks21:43
clarkbI opened jeblair's paste and was really confused about why mrodden was having trouble with endpoint not found21:43
clarkbmrodden: is there a stale pbr install?21:44
clarkbmrodden: I am not sure what to check next now that you have pip and setptools happy21:44
mroddenyeah let me double check21:44
jeblairpleia2: ok, ready for that call when you are21:44
mroddenpretty sure i got all of them...21:44
*** adalbas has quit IRC21:46
_david_mordred, you didn't mention, that i turned that plugin as pure core feature too, ... he didn't react to my question on dev ML, though21:46
mroddenclarkb: i can't find any other pbr's, i removed the /opt/stack/nova/pbr-* egg and used pip uninstall to get rid of any others21:47
mroddenand then pip installed a new one and i get the same error21:47
clarkbmrodden: pip installed pbr?21:49
openstackgerritDoug Hellmann proposed a change to openstack-dev/pbr: Do not pass unicode where byte strings are wanted
mroddenpip install pbr21:49
clarkboh you know this may be that lazy loading setuptools over distutils bug that clayg found21:49
mroddenis there a workaround?21:50
clarkbmordred: did you ever get a chance to look at that?21:50
clarkbmrodden: when I thought I ran into it I installed different versions of setuptools until it worked. I think 0.7.8 worked for me21:50
mroddeni'll give that a shot21:52
* mordred reading scrollback21:52
clarkbmordred: jeblair fungi does the current version look good? I will send it out if there are no more comments21:54
mordredjeblair: I thnk that mfick's suggestion of a new approach is doable, but as _david_ says, it's not likely to land for 2.8. _david_'s patch to 2.8 is pretty small - should we consider carrying _david_'s patch and using his plugin for 2.8, and then working with mfick to get a genericized version of that patch for the next gerrit release and migrate _david_'s patch to use that new interface when it's21:55
mordredjeblair: or would you rather wait? or would you rather think about this later?21:55
_david_mordred, that would be my suggestion.21:56
_david_mordred, jeblair we are talking more or less about 3 lines, that were approved by sop21:56
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Correct the default cacti tree name
clarkbmordred: don't forget zaro's patch, we would need it too21:57
mordredclarkb: we would?21:57
clarkbmordred: yes _david_'s plugin relies on the ACL files to determine who can WIP21:57
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Fix tripleo test cloud config
clarkbzaro's patch gives us the ability to give the right people WIP access21:57
_david_clarkb, mordred zaro's patch is going to be merged into 2.821:57
clarkboh cool21:57
_david_jep, we have a go from sop on that21:58
*** basha has quit IRC21:58
fungiclarkb: announcement lgtm21:58
mordred_david_: really, it seems stalled on a -1?21:58
mordred_david_: or am I looing at the wrong patch?21:58
mordreddoes zaro need to do something?21:58
_david_mordred, nope, you don't21:58
clarkbfungi: thanks21:58
mroddenclarkb: no dice with 0.7.821:58
mordredok. awesome21:58
mroddensame error message...21:58
_david_mordred, what mfick want there is just bool shit21:58
clarkbmrodden: I am not sure what needs to be done next then21:59
jeblairmordred: i think i'm okay with that; note that i'm not in the market for big changes until after the summit though; but we could start preparing.21:59
mordredjeblair: totally. I did't mean right now - I just meant as an approach21:59
mroddenwould my version of distribute matter?21:59
clarkbmrodden: no distribute is replaced by setuptools >= 0.721:59
mordredmrodden: ok. sorry, I'm a little lost on your scrollback - can you paste me your error?21:59
clarkband in theory that went in well because pip isn't completely falling over21:59
jeblairpleia2, lifeless, mordred, clarkb, fungi: nodepool is building images on tripleo now22:00
mroddenis this bad?22:00
mroddenopenstack@locale-dev:/opt/stack/nova$ pip list | grep distribute22:00
mroddendistribute (0.7.3)22:00
pleia2jeblair :D22:00
_david_<mordred> does zaro need to do something? => he should wait, but as i see things, the previous ps (9) is going to be merged22:00
jeblairclarkb: i have brainspace for etherpad-dev now; how may i help?22:01
mordred"Running oslo.config-1.2.1/ -q bdist_egg --dist-dir /tmp/easy_install-k2lplJ/oslo.config-1.2.1/egg-dist-tmp-r6Hxln"22:01
fungijeblair: awesome!22:01
mordredmrodden: you are running what command?22:01
mroddensudo python develop22:01
mroddenin /opt/stack/nova22:01
mordredok. can you try something to humour me?22:01
jeblairclarkb: i'm reading 50700 now22:02
mordredmrodden: try running "sudo pip install -e ."22:02
clarkbjeblair: also read
mroddenwhy did that work...22:02
_david_jeblair, <jeblair> [...] ; but we could start preparing. ... what do you mean by that?22:02
mroddenSuccessfully installed greenlet oslo.config nova22:03
mroddenCleaning up...22:03
mordredmrodden: because easy_install evil things - I'm still trying to find the simple reproduction so that I can fix it22:03
jeblairclarkb: so one question about the tuning: have you maxed out apache while load testing?22:03
mordredmrodden: in general, pip install -e . is exactly the same as python develop22:03
mordredmrodden: except a bit more resilient22:03
clarkbjeblair: not at the current levels, I hit the MaxClient warning at 102422:03
clarkband definitely hit errors when it was at the default of 15022:03
mordredmrodden: I am still trying to figure out why develop fails _sometimes_ (it's not consistent)22:03
jeblairclarkb: maxclient warning?22:04
mroddenyeah and now 'sudo python develop' works fine too22:04
mroddenits kind of a noop i'm sure22:04
mordredmrodden: it's epic failure combinations22:04
mordredclarkb: ^^22:04
mordredclarkb: workaround if this comes up again22:04
clarkbjeblair: Tue Oct 08 21:37:34 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting22:04
jeblairclarkb: (since it's the thing that will limit connections, i want to make sure that if we get a higher load than expected, we don't cause the server to swap or something -- so we want to make sure that the apache levels are _low_ enough to protect the server)22:04
*** tvb|afk has quit IRC22:05
clarkbjeblair: ok, let me spin up a ton of clients and see how apache deals with it22:05
*** dafter has joined #openstack-infra22:06
*** gyee has joined #openstack-infra22:06
*** weshay has quit IRC22:06
jeblairclarkb: cool thx.  it does at least seem like they are high enough and the server is provisioned in such a way to deal with our expected load, which is good.22:06
*** emagana has quit IRC22:06
fungiand i looked over the graphs in cacti for the new etherpad-dev. not too frightening when it was getting blasted22:07
*** emagana has joined #openstack-infra22:07
jeblairfungi: excellent22:07
mroddenthank mordred and clarkb for the help22:07
fungialso, the last couple of new nodes were not getting automatically added to cacti's all hosts tree. fixes (i manually added nodepool and etherpad-dev earlier)22:08
jeblairclarkb: tuning and announcement look good.22:09
jeblairclarkb: we may want to check on the memory usage after a week or so and consider setting maxrequestsperchild in case there's a memory leak or something.22:10
mordredmrodden: any time! sorry for the derpy setuptools stuff22:10
jeblairclarkb: hopefully that's unlikely since we're just proxying.22:10
clarkbjeblair: according to netstat I have 6k connections, according to free 'Mem:          1995       1408        586          0        119        433', and ps says 66 apache processes22:10
clarkbno swapping22:10
*** dafter has quit IRC22:10
jeblairclarkb: shouldn't you only have 2048 connections?22:11
mroddenmordred: np, at least i can get to hacking on this other issue now :)22:11
*** saper has quit IRC22:11
jeblairclarkb: is that counting apache <-> node.js connections as well?22:11
jeblair_david_: i meant that we can start testing 2.8 and your plugin on, and the upgrade process, etc.22:12
clarkbjeblair: I am only counting connections from my test box, I short for at least 2048, but I am pretty sure more than one connections is needed per client then they start failing and apache can't handle them all and queueing happens22:12
_david_jeblair, fine, let me help with this22:12
clarkbjeblair: but this should show what happens when apache is doing all it can22:12
jeblairclarkb: ox cool.22:12
*** saper has joined #openstack-infra22:13
jeblair_david_: will do, thanks :)22:13
*** jergerber has quit IRC22:13
*** thedodd has quit IRC22:13
jeblairclarkb, mordred, fungi:
mordredjeblair: neat!22:14
clarkbjeblair: it very clearly isn't able to handle 2048 concurrent users, I think ~600 is probably a closer guestimate22:14
pleia2ok, I want to test this tripleo job22:14
clarkbjeblair: which I think should be plenty22:14
jeblairpleia2: let me check on the image22:15
mordredjog0, dhellmann, sdague: ^^ check out the above link22:15
fungiyay pootle!22:15
jeblairmordred: okay, before you get too crazy...22:15
mordredjeblair: blast. I was going to announce to openstack-dev right now ...22:15
jeblairi'd rather the i18n team not get the idea that we just switched to pootle....22:15
dhellmannwow, the UK translators really need to step up their game22:16
jeblairwhat i'm trying to do, but can't quite seem to type fast enough22:16
jeblairis set up an instance so that the i18n team can poke at it and we can answer any questions before the design summit22:16
jeblairso that we can have a productive conversation about what platform to use22:17
clarkbjeblair: ++, maybe we can get jenkins jobs for one project that sync to it as well22:17
jeblairso i wanted to get mostly clarkb's opinion on if that's enough to have such a conversation, or if we need to do more...22:17
jeblairclarkb: but i don't really want to build out all the infra -- i just want to be able to know if it will meet our needs and be worth building out the infra22:18
jog0mordred: nice our persian is really good it looks like22:18
clarkbjeblair: I see, I think it is definitely a good start. But the i8ln team should provide feedback as they have to interact with it more typically22:18
jeblairjog0: you're probably looking at the sample projects.22:18
jeblairclarkb: yeah, that's the next step.22:18
jog0jeblair: too bad22:19
mordredjeblair: by "build out all the infra" - is one of those things the git/gerrit integration and how the flow back and forth works?22:19
fungii8ln is a mash-up of l10n and i18n?22:19
jeblairmordred: yep22:19
mordredjeblair: (also, I like that it has a "Translate offline" action that's easily found - in case people like working with po files)22:19
jeblairmordred: i gather it will pack up zip files for you22:19
clarkbit allows translating in browser whcih is a feature that folks wanted iirc22:20
mordredjeblair: k. I _think_ the git/gerrit integration shouldn't be necessary to figure out, beause I don't think the current teams interact with that much22:20
jeblairmordred, clarkb: i loaded in nova; should i also load in horizon?22:20
mordredjeblair: yes22:20
*** rfolco has quit IRC22:20
clarkbjeblair: yes because horizon folks are very good at it22:20
mordredjeblair: I think gabrielhurley has been on top of it before22:20
mordredand what clarkb said22:20
clarkbfungi: I probably just typed a tyop22:20
jeblairmordred, clarkb: is there something i should know about it?22:20
clarkbjeblair: about horizon? they have mutliple translation domains and extracting messages isn't magical with babel22:21
clarkbthey do not use babel because something, maybe new babel will fit their needs though22:21
jeblairclarkb: because django, i think.22:21
jeblairclarkb, mordred: their locale directory looks normal though.22:22
mordredjeblair: yeah - I think our import/export process is the same, yeah clarkb?22:22
jeblairclarkb, mordred: does that seem right?  i'm not really conversant with what's different in their approach.22:22
clarkbmordred: no it is completely different22:22
clarkbthey use gettext proper and extract into mutliple domains22:22
*** ryanpetrello has quit IRC22:23
mordredclarkb: what does babel get us again?22:23
clarkbmordred: magical integration22:24
jeblairalso, there is openstack_dashboard/locale and horizon/locale22:24
* mordred is wondering if pbr could grow understanding of what's needed there enough to handle horizon and the rest sanely22:24
clarkbwe could use gettext proper everywhere, but you have to teach it stuff and run it with the right options and so on22:24
clarkbBabel makes it easy when dealing with python memes22:24
jeblairmordred: is horizon still two projects in one repo?22:24
jeblairclarkb: how do we handle that with tx?22:24
*** _david_ has left #openstack-infra22:24
clarkbjeblair: we don't22:24
clarkbhorizon is still doing it all by hand22:24
jeblairthen i'm not sure i'm interested in doing anything with horizon in pootle at the moment.22:25
clarkbthats fair22:25
mordredwell, there is a questoin though22:26
mordredbecause horizon doing it by hand may be using the tx tools for upload/download?22:26
mordredand we may need to figure out what "doing it by hand" looks like?22:26
jeblairmordred: oh, they are using transifex?22:26
clarkbhorizon uses transifex, but the extraction and merging of translation files is manual22:27
clarkbnot sure how they upload/download22:27
mordredso we might want to manually import their files, and then let gabriel poke with that their manual process might need to look like22:27
mordredhorizon, btw, is by far the most different openstack project22:27
mordredit's on my todo list to get my head wrapped around what they do22:28
*** jerryz has joined #openstack-infra22:29
jeblairmordred: so i can set up two projects, horizon and openstack_dashboard, and deep-link into the respective locale dirs22:29
jeblair(this is just reinforcing the idea that they are, in fact, two projects in my mind)22:30
jeblairmordred: my guess is that the git workings will be okay since there shouldn't ever be any merge conflicts or anything between those two projects.22:30
clarkbjeblair: correct as the .pot and .po files should always be under separate dirs22:31
* mordred wonders if it would be worth re-addresing the issue of openstack_dashboard v. horizon22:31
mordredI've never been able to figure out the separation - but it seems important to them22:32
harlowjaah, good ole horizon22:32
harlowjaone of those most hard projets to build22:32
mordredso I'd love to understand better22:32
jeblairmordred: we should do that every summit as long as they are conjoined22:32
mordredsubmitting a talk now...22:32
mordredalready one up22:33
mordredHorizon is a framework to build UIs using restful services.22:33
mordredOpenStack Dashbaord is just a reference implementation.22:33
jeblairmordred: sounds like lib/server split to me22:34
mordredI agree22:34
mordredI have commented on the sessoin22:34
mordredhere's the problem ...22:35
mordred(with the text of the thing)22:35
jeblairmordred, clarkb: you are superusers now22:35
mordredis that "horizon" is what people call what I believe is actualy "openstack_dashboard"22:35
clarkbmordred: correct22:35
mordredand I do not believe that anyone calls what is actually horizon anything22:35
*** prad has quit IRC22:35
clarkbjeblair: can be used for CLA type enforcement?22:36
jeblairclarkb: good question, haven't looked into it yet.22:37
*** thomasm has quit IRC22:37
*** markmc has quit IRC22:37
jeblairi added horizon and openstack_dashboard as separate projects.  it might also be worth trying to add them as one project just to see what happens.22:38
*** jecarey has quit IRC22:38
mordredclarkb: I have added a legal page22:38
dansmithgerrit is dead?22:38
dansmithes muerto?22:38
mordreddansmith: nope22:38
mordreddansmith: I was just angry22:38
jeblairclarkb, mordred: fyi -- adding a project involves local filesystem stuff.22:38
dansmithmordred: you DDoSed it?22:38
dansmith(seems dead from here, FWIW)22:38
mordredthe legal page is now active and it is and set to display on registration22:38
mordreddansmith: oh!22:38
mordreddansmith: I thought you were referring to my rant from earlier22:39
dansmithothers reporting the same22:39
mordredI agree. it seems unhappy to me too22:39
jeblairmordred: i have disabled registration due to openid, so it's possible that may not have an effect22:39
mordredI can ssh into the box22:39
dansmithsomeone just said "I was about to +2 all yer patchez, but gerrit is dead"22:39
mordredtop - 22:39:52 up 146 days,  3:16,  3 users,  load average: 310.95, 271.06, 215.22:39
clarkbload average: 309.21, 269.36, 214.57 from review22:39
mordredthere are a LOT of git processes22:40
clarkbsomething is hitting the local git mirror22:40
*** sdake_ has joined #openstack-infra22:40
*** sdake_ has quit IRC22:40
*** sdake_ has joined #openstack-infra22:40
hub_capis review.o.o down?22:41
clarkblooks like our slaves22:41
hub_capits up for me now22:41
mordredhub_cap: load 30022:41
clarkbdid we make a change to d-g to make d-g slaves hit gerrit proper?22:41
hub_cap300 is good right? it means youve got enough exp to lvl soon22:42
jeblairhypothesis: nodepool built a new image22:42
mordredjeblair: that is a good hypothesis22:42
mordredjeblair: are you thinking it built an image with no repos on it?22:42
jeblairnodepool is in an image-building loop22:43
clarkblots of 15/8 IPs doing GETs on /p/*22:43
jeblairi will remove tripleo config from nodepool22:43
*** mrodden has quit IRC22:43
dansmithmaybe we should plug the box running gerrit into 220V22:44
*** datsun180b has quit IRC22:44
jeblairi have stopped nodepool and will delete images now22:45
mordredjeblair: please let me know if there are things I can do to be helpful22:45
jeblairmordred: can you look at devstack-gate runs and verify ?22:45
clarkbjeblair: definitely much fewer 15/8 hosts hitting apache on review.o.o now. Load is falling slowly22:46
mordredjeblair: yes. verify what?22:46
jeblairmordred: hypothesis22:46
*** blamar has quit IRC22:46
jeblairclarkb: i saw errors in the logs about too many keypairs22:47
*** pentameter has quit IRC22:47
jeblairclarkb: i think there may be manual cleanup required22:47
jeblairclarkb: can you look into deleting keypairs?22:47
mordredjeblair: devstack-gate runs seem to be operating as normal22:47
clarkbjeblair: yup, this would be for hpcloud, rax or tripleo or all three?22:47
clarkbjeblair: or jenkins?22:48
clarkbtoo many places for keypairs to exist22:48
jeblairmordred: hypothesis negated?22:48
mordredon the ones that were already running. there are no new runs running, I'm assuming because nodepool is down22:48
jeblairclarkb: i think just hpcloud?22:48
clarkbjeblair: ok22:48
jeblairmordred: er, wait22:48
*** julim has quit IRC22:48
jeblairmordred: so you looked at the logs and discovered that the devstack-gate jobs were NOT hitting gerrit?22:48
mordredjeblair: no. that is not what I have verified. sorry, one sec22:49
mordredthis is not what I'm looking for currently, but as a note to fix soon:22:50
mordredgit_clone /opt/stack/new/requirements master22:50
clarkbthere are a bunch of nodepool keypairs under hpcloud az1 but no az2 or az3. Looking at az1 more closely now22:53
clarkbkeypair-show doesn't really show us any special info22:54
clarkbshould I go ahead and delete the nodepool keypairs in az1?22:54
mordredjeblair: is the most recent run of devstack-gate to report and upload all logs22:54
jeblairclarkb: yes please22:54
*** dizquierdo has joined #openstack-infra22:54
mordredjeblair: I'm seeing origin setting to git.o.o and zuul fetching in that file, but no review.o.o stuffs22:55
jeblairmordred: it seems to have git repos22:55
jeblairmordred: gah, nodepool doesn't log the ip22:56
mordredjeblair: what did you mean by "nodepool is in an image-building loop"?22:56
jeblairmordred: it was continuously building images22:56
mordredjeblair: it's never the thign we need in the logs :)22:56
jeblairmordred: that node was spun up a long time ago22:57
clarkball but one deleted. I left in because none of the others had suffix22:58
jeblairclarkb: you can delete it too22:59
clarkbjeblair: ok22:59
fungigah, i step away to make toast, and come back to disaster23:00
clarkbthat keypair doesn't want to delete with novaclient 'ERROR: Keypair nodepool-precise-1376001841.template.openstack not found for user 37577772860620 (HTTP 404)' looks like it may be truncating the keypair name. Going to try it through the webui23:01
fungistill need help?23:01
jeblairclarkb: oh, right, nevermind about that.  you can't delete it.23:01
jeblairclarkb: bug in hpcloud.  i opened a ticket years ago.  they didn't care.23:01
lifelessjeblair: cool23:01
clarkbjeblair: gotcha23:02
lifelessjeblair: where can mortals observe nodepool status?23:02
clarkbit did error in the webui23:02
hub_capfungi: u need toaster as a service23:02
jeblairlifeless: they can't23:03
fungihub_cap: or just a usb toaster in my lab23:03
hub_capu know fungi u coudl probably just have toaster in your lab23:03
fungion a ups23:04
hub_capthx for getting things back up-n-runnin guys (/me assumes things are back online)23:04
hub_capfungi: gotta have that 30min to unplug the toaster23:04
lifelessjeblair: ok; so - did it succeed, or do we need to tweak it? are we able to test our primitive experimental job?23:04
fungihub_cap: yeah, load on review.o.o is back down to normal levels now23:04
mordredlifeless: we're towards the end of a bit of an incident23:05
mordredlifeless: at the moment, short answer - no23:05
mordredlifeless: long answer, let us get back to you ina bit23:05
mordredlifeless: (for context, see recent backscroll)23:05
*** rcleere has quit IRC23:07
lifelessmordred: ack23:07
jeblairmordred: i'm not sure that was us23:08
jeblairmordred: you haven't found supporting evidence yet, and i've started spot checking ip addresses that were hitting us right after i stopped nodepool ---23:09
jeblairmordred: that means that nodepool should not have deleted them, which mean they should be in the list of servers23:09
jeblairmordred: but they are not23:09
mordredjeblair: uhm.23:09
mordredjeblair: but why did it stop when you killed nodepool?23:10
jeblairmordred: oh, you know what they could be --23:10
jeblairmordred: they could be the image builds themselves..23:10
mordredjeblair: yes. because of the image build loop23:11
clarkb15.185.100.98 appears to be one culprit in apache logs23:11
clarkbit unpings23:12
jeblairi found what sent it into a loop23:12
jeblairand pabelanger already fixed it :)23:12
jeblairokay so here's the summary:23:14
jeblairsince i removed the tripleo config, we shouldn't hit this code path, so i think we're unlikely to have a recurrance when we restart23:14
*** dafter has joined #openstack-infra23:16
mordredk. but - you also have merged the fix from pabelanger which also should fix the immediate thing we saw, for whever we decide to re-add that config23:17
jeblairmordred: well, i approved it but it's waiting on a dependent23:18
jeblair(which is unfortunate, it's actually an independent change)23:19
mordredjeblair: perhaps we should rebase the approved patch on master? or just wait until we get feedback from pabelanger?23:19
jeblairmordred: probably rebase, but not now23:19
jeblairi'm doing manual cleanup of some nodes now23:19
jeblairshould be able to restart in a few mins23:19
mordredjeblair: although - looking at the patch and the author - I think it's probably safe to say that if there are more sqlite issues, he'll show up with patches23:19
mordredjeblair: kk23:19
*** mriedem has joined #openstack-infra23:20
*** dafter has quit IRC23:21
mordredjeblair: commented as above on the review itself23:21
jeblairstarting nodepool23:24
jeblairinadvertent test of nodepool bursting ^23:24
jeblairseems to work :)23:25
jeblairi think things are running now.  nodepool host is still in a manually configured state so it doesn't have the tripleo config23:25
jeblairpuppet is still stopped there23:26
clarkbjeblair: the log messages about ASR for X (out of Y) are a bit confusing23:26
clarkbjeblair: it looks like we are spinning up more nodes than necessary23:26
clarkbfor example 10 out of 19, 10 out of 9, 9 out of 0 for devstack-precise23:27
jeblairclarkb: ignore "out of" in the grant section, i don't think that's meaningful23:27
jeblairclarkb: (ASR should be converted to a different object after the grant happens)23:28
jeblairnodes are successfully coming online now23:29
jeblairi think we're good now.  and i have to run23:29
mordredjeblair: have fun!23:29
mordredlifeless: so, now that we're out of the woods... it may be a little bit longer before you have experimental queue23:30
mordredlifeless: we're going to need to land pabelanger's patch23:30
mordredlifeless: and then reenable your config23:30
mordredlifeless: which is going to want a jeblair around23:30
lifelessFWIW this is why I'm pushing on it because I expect teething problems :)23:30
pleia2woo, I broke everything23:33
*** dcramer_ has joined #openstack-infra23:37
*** dizquierdo has left #openstack-infra23:38
wenlockasking pip experts, is there any way to use pip behind a proxy besides using --proxy option with pip?... I'm guessing the pip module in ci project doesn't work behind proxies....correct?23:39
*** fifieldt has joined #openstack-infra23:39
clarkbpuppet's pip module does not work behind proxies23:41
clarkbmordred: ^23:41
jog0gate is looking strong these days:
jog0really low failure rate23:43
clarkbhmm etherpad-dev is in that funny state again where I can have mutliple clients on one pad and not see pad content update across them, but IM works23:44
jog0mtreinish sdague ^^23:44
clarkbanyone else want to confirm that they see the same thing on etherpad-dev?23:44
sdaguejog0: yeh, I was noticing the recheck resets are trending down nicely23:45
sdagueclarkb: url?23:45
clarkbsdague: is the pad I am using23:46
clarkbI think it may be triggered by something the stress test is doing. Last time I ran into it I restarted etherpad and it was happy again23:46
jog0sdague: yeah there less cases of someone having to run recheck 3 times in a row now23:46
wenlockclarkb: i started a new test pad on  i see it between ff and chrome, not updating23:47
sdagueclarkb: yeh, I've seen that on our pad from time to time23:47
sdaguea restart usually helps23:47
clarkbwenlock: that is what I see too, thanks for checking23:47
clarkbsdague: interesting, and idea if there is an upstream bug? I am looking through their bug list now23:47
sdaguewhen it doesn't, I just delete /opt/etherpad-lite and let it repuppet23:47
clarkbsdague: you are filling me with confidence :)23:48
sdaguenot that I know of. My experience is the irc channel is the place to start asking questions23:48
sdagueI really like etherpad... however it's ops story is terrible23:48
sdaguethe logs provide basically nothing useful to debug with from what I can tell23:49
*** torgomatic has quit IRC23:49
sdagueclarkb: though I haven't seen this in a while, what version are you at?23:49
clarkbsdague: I am using master I think (I thought I checked it included your patch)23:50
sdagueyou want the tip of develop23:50
clarkbya the logs are not very helpful23:50
mordredclarkb: that is correct23:50
*** sarob_ has quit IRC23:50
clarkbsdague: ok let me switch to that now really quick, then I am going to send my message about the upgrade23:50
mordredwenlock: so... the thing I was about to do to solve the puppet+pip+proxy issue23:51
*** sarob has joined #openstack-infra23:51
mordredwenlock: was to run a copy of devpi behind the firewall23:51
mordredwhich CAN deal with being behind a proxy23:51
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Make discoverable
mordredand then configure everything in puppetland to use that23:51
jog0I have a hunch I messed ^ up23:51
jog0as I don't know js23:51
wenlockmordred:  i was just looking at the module, why couldn't we just make it smarter and find http_proxy env and use it23:52
mordredwenlock: well...23:52
mordredwenlock: we could23:52
mordredPackage <| provider=='pip' |> {23:52
mordred  install_options => "--proxy="23:52
mordredis a thing I have in the internal one23:52
mordredand it WOULD work23:52
wenlockmind if i code it up tonight and give it to you?23:52
mordredinside of the upstream puppet code, puppet decided to use direct ruby xmlrpc calls23:53
mordredwenlock: god no!23:53
mordredthat would be AMAZING23:53
mordred(also, you might need to tell me how to install/use it)23:53
mordredbut I'm SO in favor of that23:53
openstackgerritClark Boylan proposed a change to openstack-infra/config: Update etherpad and etherpad puppet manifests.
openstackgerritClark Boylan proposed a change to openstack-infra/config: Tune apache for etherpad lite.
mordredwenlock: you may become my favorite person...23:53
clarkbI am applying that change to etherpad-dev and etherpad now23:54
clarkbwill hit it with load testing again tomorrow23:54
sdaguejog0: commented, and yes, it needs fixes23:54
jog0sdague: thanks23:54
sdaguebut I would git blame to figure out who thought 2 statements per line was a good idea :)23:54
*** sarob has quit IRC23:55
jog0git blames ttx23:55
clarkbapparently changing the ref in vcsrepo doesn't do anything23:56
openstackgerritJoe Gordon proposed a change to openstack-infra/config: Make discoverable
* clarkb does it the sdague way23:56
sdagueyeh, vcsrepo has issues23:57
fungivcsrepo has broken23:57
sdagueclarkb just think of it as devops :)23:57
sdaguewhen in doubt, just delete everything, and let puppet rebuild23:57

Generated by 2.14.0 by Marius Gedminas - find it at!