Wednesday, 2014-09-03

clarkbjeblair: tl;dr is exim said no such user for third-party-request@lists.openstack.org00:00
*** praneshp_ has joined #openstack-infra00:00
clarkbjeblair: that shows up in the exim fail log. There is no correspodning error in the mailman errorlog00:00
*** unicell has joined #openstack-infra00:00
*** emagana has quit IRC00:00
fungithe suffix addresses working is sort of odd since the mailman router configuration has them as optional00:01
*** craigbr has joined #openstack-infra00:01
*** emagana has joined #openstack-infra00:01
anteayajeblair: thanks00:01
*** praneshp has quit IRC00:03
*** praneshp_ is now known as praneshp00:03
fungia dump of the pickle file for it looks sane00:03
*** emagana has quit IRC00:05
jeblair2014-09-02 23:33:37 1XOxZx-0000d0-0T ** | /var/lib/mailman/mail/mailman post third-party-request <> R=system_aliases T=address_pipe: Child process of address_pipe transport returned 2 from command: /var/lib/mailman/mail/mailman00:07
*** otherwiseguy has quit IRC00:08
fungiSep 02 16:56:53 2014 (25772) No such list "third-party-reques":00:09
fungi(from /var/log/mailman/error)00:09
anteayasomething truncated the name?00:10
fungieh, no that may not be configuration related00:10
clarkbfungi: the timestamps don't quite line up on that either00:10
fungithe timestamp makes it look like it was probably someone's request00:10
*** otherwiseguy has joined #openstack-infra00:11
fungiso... the times i see in the exim rejectlog are *before* puppet created the list at 23:49:08 (per syslog)00:12
funginope, false alarm00:12
fungiwe rerun that ensure->created on every puppet run00:12
openstackgerritlifeless proposed a change to openstack-infra/reviewstats: Tripleo project updates.
*** gokrokve has joined #openstack-infra00:12
fungilist has been created there for a good long while00:13
clarkbfungi: huh that is probably a puppet resource bug then00:13
clarkbbut shouldn't affect an existing list. it hasn't in the past that we are aware of00:13
fungii think it's just noise from the mail alias configuration in the mailman module00:14
jeblairexim -d -bt third-party-request@lists.openstack.org00:14
jeblairrun that and you'll have the answer00:14
jeblair--------> mailman_router router <--------00:14
jeblairfile check: /var/lib/mailman/lists/${lc:$local_part}/config.pck00:14
jeblairexpanded file: /var/lib/mailman/lists/third-party/config.pck00:14
clarkbexpansion isn't working right00:15
jeblairit is00:15
clarkbbecause -request is special?00:15
fungipoor name choice there :/00:15
jeblairyep.  it's one of the special mailman addresses00:15
jeblairlike bounces, confirm, jon, leave owner, subscribe, unsubscribe, admin...00:15
anteayaso the way forward?00:16
jeblairprobably rename the list, yeah00:16
anteayawould -requests work?00:16
anteayadifferent enough?00:16
lifelessalso file a bug on mailman, it shouldn't have let that list be created00:17
jeblairi mean, it's exim, so we _could_ work around it, but i don't think this is worth invalidating mailman documentation, etc.00:17
anteayaor should we be more obvious00:17
anteayawe only have 14 subscribers00:17
*** tomoe_ has joined #openstack-infra00:17
anteayawe can get them moved over00:17
*** otherwiseguy has quit IRC00:17
*** terryw has joined #openstack-infra00:17
jeblairi expect we'll probably stop puppet, merge rename change, manually rename on disk, start puppet00:19
clarkbyes, otherwise you wil lend up with two lists00:19
anteayado I need to get the subscribers to resubscribe?00:19
jeblairanteaya: nope00:20
clarkbpuppet can delete though if you want to ensure => absent on the old one00:20
*** erlon has quit IRC00:20
pleia2hah, oops00:20
jeblairclarkb: oh, maybe we should do that instead?00:20
openstackgerritA change was merged to openstack-infra/config: Use zm04
anteayajust change the link on third-party.rst00:20
fungiokay, once puppet applies that ^ to zuul, zm04 should start picking up merge-check work00:20
*** chuckC has quit IRC00:20
anteayawhat have we agreed to rename the list?00:20
jeblairi'm okay with either way, but maybe the delete/recreate method is less error-prone?00:21
jeblair(either way, i'm thinking we do it tomorrow)00:21
clarkbI know how to review and babysit delete/recreate but renamign a list and any associated cleanup mailman side would be foreign to me00:21
clarkbjeblair: ++ I need to walk home shortly00:21
fungidelete it from puppet configuration (not ensure->absent), rename on disk, readd to puppet with correct name?00:21
clarkbfungi: that will work too00:22
jeblairfungi: oh that's a cleaner rename way, yeah00:22
fungiprevents stopping puppet too00:22
*** sballe_ has quit IRC00:22
jeblairi'm okay doing the rename on disk and troubleshooting if it goes awry00:22
anteayaI'm about to offer a patch to the docs00:23
anteayajust need to know what we are renaming it00:23
fungianyway, i'm knocking off for the evening, other than keeping an eye on zm04 for the next few minutes00:23
jeblairanteaya: -requests should be fine00:23
jeblairanteaya: would you also like to write the list changes?  the first one should just remove the list definition entirely (ie, just delete it from the file).  then the second should add it back, but with the new name.00:24
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Edit the name of the third-party-requests ml
anteayajeblair: I can try00:27
openstackgerritJay Faulkner proposed a change to openstack-infra/config: Publish individual files in IPA post job
anteayaI'll work from pleia2's patch:
JayFfungi: sdague: clarkb: this is my first attempt at something to make the IPA devstack stuff more palatable, I've laid out a general plan in the commit message, please let me know what you guys think in gerrit :) CC: jroll00:29
*** sballe has joined #openstack-infra00:31
*** gokrokve_ has joined #openstack-infra00:33
*** gokrokve has quit IRC00:34
*** yamahata has joined #openstack-infra00:34
*** baoli has joined #openstack-infra00:35
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Don't modify status widgets outside of main thread
*** yamahata has quit IRC00:36
*** bhuvan_ has quit IRC00:36
*** mrodden has quit IRC00:36
*** yamahata has joined #openstack-infra00:37
*** mrodden has joined #openstack-infra00:37
*** craigbr has quit IRC00:37
*** nijaba has quit IRC00:38
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list
openstackgerritlifeless proposed a change to openstack-infra/reviewstats: Make pdb debugging of openreviews possible.
anteayahow do I make a dependency of 118509?00:44
anteayasince it should go in after00:44
lifelessgit review -d 118506; git review -x 118509; git review. IIRC.00:44
*** ominakov has joined #openstack-infra00:44
jeblairanteaya: but you still want 08 in there, so you should add it into the line lifeless suggests: -d 06, -x 08, -x 09.00:47
anteayashould I be in the branch with 09 in it or master?00:48
anteayaor will git review -d take me to a new branch with 506 anyway00:48
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list
jeblairanteaya: new branch with 50600:51
anteayakk, well they are all together then00:51
anteayaI had wanted 06 at the end but it is at the front00:51
anteayaoh well00:51
*** gokrokve_ has quit IRC00:51
anteayadoes't matter00:51
anteayaI'll know for next tiem00:51
jeblairoh, heh.  you can do -d 08, -x 09, -x 06.00:52
anteayathis is new:
anteayagit review blew up on me00:54
anteayamy choices are git commit --allow-empty or git reset00:55
*** mjturek has joined #openstack-infra00:56
jeblairanteaya: oh, it's probably because starting with 08 meant it also pulled in 06 since it already depended on it..00:59
anteayaI think so00:59
anteayareading about git reset options00:59
*** terryw has quit IRC00:59
*** r-daneel has quit IRC01:00
*** pcm_ has quit IRC01:00
jeblairanteaya: git reset; git checkout master; git checkout -b somenewbranch01:00
jeblairanteaya: then do the git review series again, except use -x for all three01:00
anteayaah okay01:00
jeblairanteaya: (that way you are cherry-picking each change in sequence, starting with master (so it won't pull in any of the other changes in the current sequence until you tell it to))01:01
*** reed has quit IRC01:01
*** marcusvrn has quit IRC01:01
anteayaah ha01:01
*** mjturek has quit IRC01:01
*** reed has joined #openstack-infra01:03
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Edit the name of the third-party-requests ml
*** mmaglana has quit IRC01:03
*** erlon has joined #openstack-infra01:04
anteayayay thats the order I wanted, thanks jeblair01:04
*** gyee has quit IRC01:04
*** lnxnut has joined #openstack-infra01:06
*** yaguang has joined #openstack-infra01:07
*** lcheng has quit IRC01:07
*** dims has joined #openstack-infra01:11
*** laca has joined #openstack-infra01:11
*** otter768 has joined #openstack-infra01:11
*** mwagner_lap has joined #openstack-infra01:14
*** lnxnut has quit IRC01:16
*** gokrokve has joined #openstack-infra01:17
phschwartzevening infra from Disney.01:17
*** bknudson has joined #openstack-infra01:18
dimsphschwartz: nice :)01:21
phschwartz:) trying to get some work in while on a family vacation. lol01:21
clarkbdoes the family know?01:21
*** dims has quit IRC01:23
*** dims has joined #openstack-infra01:23
*** marcoemorais has quit IRC01:23
phschwartzYes, Wife is watching TV in bed next to me01:23
*** marcoemorais has joined #openstack-infra01:24
* nibalizer casually wonders if disney has an openstack01:24
clarkbnibalizer: they run swift01:25
phschwartznibalizer: In ATL I talked to a few of their engineers. The run a Nova setup and a Swift setup.01:26
*** dims_ has joined #openstack-infra01:26
*** zz_dimtruck is now known as dimtruck01:26
dims_whoa! loooooooong gate01:27
*** dims has quit IRC01:28
*** mestery has quit IRC01:31
*** mestery has joined #openstack-infra01:31
*** sballe has quit IRC01:32
*** mestery_ has joined #openstack-infra01:35
openstackgerritSteve Kowalik proposed a change to openstack-infra/config: Move os-{apply,collect}-config to python3-jobs
*** trinaths has quit IRC01:35
openstackgerritJia Dong proposed a change to openstack-infra/config: Add Tricircle project
*** camunoz has joined #openstack-infra01:36
lifelessclarkb: re
*** marcoemorais has quit IRC01:36
lifelessclarkb: what other subunit logs?01:36
clarkblifeless: the ones we grab for the unittests, I think they have the same issue01:36
clarkblifeless: I can write up the fix for them though01:36
*** camunoz has quit IRC01:37
*** mestery has quit IRC01:37
nibalizerphschwartz: clarkb sweet01:37
lifelessclarkb: I thought that was the one I patches01:38
lifelessclarkb: oh, no I see - yes please do01:38
lifelessclarkb: I'm || close to turning on subunit-v2 for the storage backend01:38
lifelessso we'll really want that changed at that point :)01:39
*** Ryan_Lane has quit IRC01:42
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Fix crash on dependency update
jeblairtricircle sounds like tripleo01:43
jeblairnot so much the project, but the name :)01:44
lifelessthree pleo's ?01:44
jeblairbeep boop bleep01:44
*** stevemar has joined #openstack-infra01:46
anteayajeblair: that's what I thought01:46
anteayasounds to me like the tried to match tripleo's name as closely as possible01:47
*** melwitt has quit IRC01:48
openstackgerritJia Dong proposed a change to openstack-infra/config: Add Tricircle project
jeblairanteaya: could be complete coincidence.  trinities and round shapes show up a lot :)01:49
*** Sukhdev has joined #openstack-infra01:50
*** gargola has quit IRC01:50
*** pcrews has quit IRC01:50
anteayaah, I had to get them to reduce the use of openstack in their commit message from 3 mentions to 101:50
anteayaI'm not feeling coincidence here01:50
anteaya3 mentions in one sentence01:50
anteayabut lifeless looked and gave me the go-ahead01:51
anteayaso *shrug* I will01:51
*** Sukhdev has quit IRC01:51
*** pcrews has joined #openstack-infra01:52
*** nosnos has joined #openstack-infra01:52
*** changbl has joined #openstack-infra01:52
*** imeerovi has joined #openstack-infra01:53
nibalizeri wish we'd called tripleo ozone01:55
nibalizerthat name is so cool01:55
anteayathat would go over well in the trademark filter01:55
nibalizeroh right01:56
anteayaI'm sure noone else has that01:56
nibalizerits all mine and your anita01:56
nibalizerwe can escape with it01:56
anteayathough I like it as well01:56
nibalizerand hold the foundation hostage for millions!01:56
anteayano no01:56
anteayaif i am escape there are horses and beaches01:56
nibalizertru dat01:56
anteayaand mountains will teepees owls and coyotes01:56
anteayaand boats01:57
anteayano hostages01:57
anteayathey just get in the way01:57
nibalizerhaha, deal01:57
*** JadnG has joined #openstack-infra01:57
* anteaya takes nibalizer's hand and skips away01:57
* nibalizer turns on happy-pop music01:57
anteayaha ha ha01:57
*** imeerovi has quit IRC01:58
*** baohua has joined #openstack-infra02:00
jeblairi switch desktops for 5 minutes and this happens :)02:00
jeblairi should switch desktops more often02:00
anteayaha ha ha02:00
*** yongli has quit IRC02:02
JadnGHello. Help please. Just create a stackforge project Tricircle. Can anyone review and approve that request. We'd like to contribute source code ASAP. Thanks02:02
*** dims_ has quit IRC02:03
JadnGThe link
*** yongli has joined #openstack-infra02:03
*** dims has joined #openstack-infra02:03
anteayahi JadnG we are reviewing your patch as fast as we are able02:04
anteayaJadnG: this is a very busy week for us in our release cycle02:04
JadnGThank you very much. It's great helpful02:04
*** baohua has quit IRC02:04
anteayaJadnG: great, you are quick at offering new patches, that helps02:05
*** zhiwei has joined #openstack-infra02:05
JadnGWe also want the tricircle source code being a candidate part of Kilo. Need discussion in the summit.02:05
JadnGThank you all a lot.02:06
*** baohua has joined #openstack-infra02:06
*** adalbas has quit IRC02:07
anteayaJadnG: ah well you are a stackforge project, if you want to be considered for incubation in kilo you need to start following the tc meetings02:07
*** dims has quit IRC02:08
zhiweiHi all, I encountered an jenkins issue in my commit, any one can help?02:08
anteayapleia2: kevin mitchell might be a good person to get some input on style guidelines:
anteayazhiwei: we don't know yet02:08
anteayazhiwei: can you post the url of the commit?02:08
zhiweiok, thanks.02:08
anteayaJadnG: start here:
*** mestery_ is now known as mestery02:09
zhiweithe buff-ruby_engine gem need ruby version >=1.9.202:09
zhiweibut the jenkins server has 1.802:09
pleia2anteaya: ooh, perhaps!02:09
anteayazhiwei: your test ran on precise02:10
*** salv-orlando has quit IRC02:10
anteayazhiwei: update your test node to run on trusty02:10
anteayaBuilding remotely on bare-precise-hpcloud-b1-1856260 in workspace /home/jenkins/workspace/gate-cookbook-openstack-common-chef-lint02:11
anteayaunless we are running trusty on nodes called bare-precise?02:11
anteayazhiwei: it needs to run on 14.04 but it didn't02:11
zhiweianteaya: please look this url:
anteayazhiwei: you need to change the test node information02:12
zhiweiI think the jenkins server has multi ruby version, but I don't why in my commit, it uses the wrong version.02:12
anteayauilding remotely on bare-trusty-hpcloud-b2-1858453 in workspace /home/jenkins/workspace/gate-cookbook-openstack-block-storage-chef-lint02:12
anteayadifferent test02:13
anteayathat one built on trusty02:13
anteayathe jenkins server has multiple test nodes02:13
anteayaand runs tests on the node type your project specifies02:13
*** dims has joined #openstack-infra02:13
*** gokrokve_ has joined #openstack-infra02:14
*** homeless has quit IRC02:14
zhiweisorry, I can understand, can you detail it?02:14
pleia2if your tests need to run on trusty, you need to specify that in the test, right now it just picks any available ubuntu node (precise or trusty)02:15
anteayawhat is the name of this project?02:15
*** tgohad has quit IRC02:15
anteayawhat a name02:15
*** nosnos has quit IRC02:16
*** gokrokve_ has quit IRC02:16
anteayaremove bare-precise ||  from that file02:16
zhiweiok, thanks.02:16
pleia2and I'd confirm with the team that is what is desired02:16
pleia2if it should run on precise, the code should be altered, not the test02:17
anteayazhiwei: oh yes, offer the patch and get it reviewed02:17
*** gokrokve has quit IRC02:18
*** stevemar has quit IRC02:18
zhiweipleia2: yes, I think I need to explore it why the code can't work on precise.02:19
*** stevemar has joined #openstack-infra02:19
anteayabecause precise doesn't have the version of ruby one of your gems needs02:19
*** chuckC has joined #openstack-infra02:19
anteayaso re-evaluate if you really need that gem, or that version of the gem02:20
*** yamamoto has quit IRC02:20
zhiweithis gem is inside in another gem, and I don't provider in the project Gemfile.02:20
*** gokrokve has joined #openstack-infra02:21
anteayazhiwei: that is a choice your team has to make02:22
*** bdpayne has joined #openstack-infra02:22
anteayazhiwei: we can't make that choice02:22
zhiweiok, I will discuss this with our team later.02:23
*** tomoe_ has quit IRC02:24
*** gokrokve has quit IRC02:25
anteayahands up anyone who still thinks tricircle is a coincidence02:27
*** imeerovi has joined #openstack-infra02:27
*** pcrews has quit IRC02:28
zhiweianteaya: please check this url:
zhiweithey have almost the same code, but the result is not same.02:28
zhiwei  the patch set1 and patch set 6 has minor changes. But the jenkins job failed in patch set 6.02:30
*** baohua has quit IRC02:31
*** baohua has joined #openstack-infra02:32
anteayaAn error occurred while installing buff-ruby_engine (0.1.0), and Bundler cannot02:32
anteayaif the mirror you were downloading your gem from had a problem during the download, then the gem couldn't be downloaded02:33
anteayaopenstack doesn't have gem mirrors02:33
anteayaso you must be installing from rubygems or somewhere02:33
anteayaso recheck the patch and run the job again02:34
anteayaokay I'm tired02:35
anteayadone for the night02:35
zhiweiok, thanks.02:36
*** amcrn has quit IRC02:38
*** mmaglana has joined #openstack-infra02:38
*** imeerovi has quit IRC02:38
*** david-lyle has joined #openstack-infra02:39
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Query projects in batches
jeblairlifeless: if you have a moment, i'd appreciate your thoughts on 105032 and 11851902:41
*** mriedem has quit IRC02:42
anteayaoh forgot to say, I am dentisting off the top of the day02:49
anteayashould be back in the afternoon some time02:49
*** imeerovi has joined #openstack-infra02:49
*** harlowja is now known as harlowja_away02:52
*** jyuso has joined #openstack-infra02:53
lifelessjeblair: I had trouble making multiple q= work02:53
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add subunit2sql-db-manage utility
lifelessjeblair: I'm fine with groups of 10, though its likely many gertty users will have more than 10 subscriptions02:54
*** yamamoto has joined #openstack-infra02:55
lifelessjeblair: reviewed02:56
*** imeerovi has quit IRC02:56
lifelessjeblair: I think its hard to choose between them - there will be cases where each is better02:56
*** dims has quit IRC02:56
*** daya_k has joined #openstack-infra02:56
lifelessjeblair: I think I found a logic bug in the batch-of-10 patch02:56
*** dims has joined #openstack-infra02:57
jeblairlifeless: oh yep, you're right about that, thanks02:58
lifelessjeblair: I suspect you're right about only getting the first 500 changes today02:59
lifelessjeblair: with my patch, but thats clearly fixable02:59
lifelessjeblair: I'm pondering whether we should have both optimisations and a heuristic to choose02:59
jeblairlifeless: the 500 limit was true even before your patch; i fixed it for the base case earlier in the series (but you can see it there)03:00
jeblairlifeless: i expect that you mostly didn't notice it even with your patch since the deltas between queries are are somewhat likely to be < 500 changes03:00
jeblair(depending on how often gertty is not running)03:00
*** KanagarajM has joined #openstack-infra03:01
jeblairor at least, the impact may have been minimal to moderate03:01
lifelessjeblair: I suspect it was in the bucket of 'thats odd, I should look into it' that I had in fact not looked into03:01
jeblairthough robbie's test would have definitely been affected03:01
*** dims has quit IRC03:01
jeblairbecause syncing many projects at once from 0 should be thousands of changes03:01
jeblair(nova currently has 650 open, neutron 527)03:02
jeblair(nova is down from >700 this weekend when i was looking into this)03:02
openstackgerritJames E. Blair proposed a change to stackforge/gertty: Query projects in batches
*** hdd has quit IRC03:07
*** erlon has quit IRC03:10
clarkbfyi if we extrapolate from we have already enqueued the last chagne that will get into oepnstack prior to feature freeze03:18
*** imeerovi has joined #openstack-infra03:18
clarkbhowever zuul only reports a 14 hour time delta03:19
clarkbI assume failed commits account for that?03:19
*** praneshp has quit IRC03:20
*** otter768 has quit IRC03:21
*** zz_naotok is now known as naotok03:22
*** imeerovi has quit IRC03:23
*** praneshp has joined #openstack-infra03:23
*** zhiwei has quit IRC03:26
*** dimtruck is now known as zz_dimtruck03:26
*** koolhead17 has joined #openstack-infra03:29
*** koolhead17 has joined #openstack-infra03:29
*** imeerovi has joined #openstack-infra03:31
*** bhuvan has joined #openstack-infra03:37
*** lttrl has quit IRC03:37
*** praneshp has quit IRC03:38
*** imeerovi has quit IRC03:39
*** lttrl has joined #openstack-infra03:39
*** SumitNaiksatam has joined #openstack-infra03:39
*** praneshp has joined #openstack-infra03:40
jyusoHi folks,I have a question:I use Gerrit trigger(not Zuul) as Jenkins trigger.If I want to generate a "testr_result.html" file after CI testing,what should I do?03:42
*** praneshp_ has joined #openstack-infra03:45
*** praneshp has quit IRC03:45
*** praneshp_ is now known as praneshp03:45
*** radez is now known as radez_g0n303:46
*** lcheng has joined #openstack-infra03:46
*** Sukhdev has joined #openstack-infra03:52
*** bdpayne has quit IRC03:53
*** bdpayne has joined #openstack-infra03:54
*** melwitt has joined #openstack-infra04:06
*** mbacchi has quit IRC04:07
*** garyh has quit IRC04:09
*** melwitt1 has joined #openstack-infra04:13
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Fix pep8 issues
openstackgerritJoshua Hesketh proposed a change to openstack-infra/jenkins-job-builder: Start libification of jjb
*** melwitt has quit IRC04:16
*** changbl has quit IRC04:24
*** zz_dimtruck is now known as dimtruck04:32
*** KanagarajM has quit IRC04:34
*** ianw has quit IRC04:37
*** ianw has joined #openstack-infra04:37
*** garyh has joined #openstack-infra04:40
*** bdpayne has quit IRC04:45
*** nosnos has joined #openstack-infra04:50
Alex_GaynorIs something broken with zuul? status. doesn't seem to be updating, the requests are just hanging04:53
jogoAlex_Gaynor: status.o.o/zuul is updating for me04:55
*** amotoki has joined #openstack-infra04:56
* jogo snipes 10523104:56
Alex_Gaynorseems to be workign now, I suppose it's all in my head04:56
jogoAlex_Gaynor: once you mentioned it, it was in my head too04:56
jogohmm I wonder if there is an automatic way to sort out when something is actually broken04:57
jogoby something I mean unit tests/pep8 failing04:57
*** harlowja_at_home has joined #openstack-infra04:59
*** ppai has joined #openstack-infra04:59
*** yfried_ has quit IRC05:00
*** dimtruck is now known as zz_dimtruck05:00
openstackgerritA change was merged to openstack-infra/config: Add zaqar to
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Fix pep8 issues
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Fix up status page dependencies
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove resolved fingerprints
stevemarzuul is getting wrecked right now :(05:10
stevemar140 in the gate05:11
*** chandankumar has joined #openstack-infra05:13
*** enikanorov__ has quit IRC05:23
*** bdpayne has joined #openstack-infra05:24
*** ildikov has quit IRC05:28
*** tsg has joined #openstack-infra05:32
*** cipcosma has joined #openstack-infra05:34
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add gerrit reviews into patchset approvals
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Allow a pipeline to specify alternative gerrit acc
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: Add support for negative requirements
*** rlandy has joined #openstack-infra05:44
*** Sukhdev has quit IRC05:52
*** harlowja_at_home has quit IRC05:55
*** Ryan_Lane has joined #openstack-infra06:01
*** bdpayne has quit IRC06:02
*** Ryan_Lane has quit IRC06:03
*** baoli has quit IRC06:08
*** imeerovi has joined #openstack-infra06:12
*** denis_makogon has quit IRC06:12
*** denis_makogon has joined #openstack-infra06:12
*** stevemar has quit IRC06:16
*** skolekonov has joined #openstack-infra06:16
*** loki184 has joined #openstack-infra06:17
*** nelsnelson has quit IRC06:17
*** k4n0 has joined #openstack-infra06:18
*** imeerovi has quit IRC06:18
*** afazekas has joined #openstack-infra06:18
*** david-lyle has quit IRC06:19
*** harlowja_at_home has joined #openstack-infra06:20
*** flaper87|afk is now known as flaper8706:30
*** liam_ has joined #openstack-infra06:30
*** liam_ is now known as Guest8123306:31
*** andreykurilin_ has joined #openstack-infra06:31
*** HeOS has quit IRC06:32
*** Guest81233 has quit IRC06:32
*** yfried_ has joined #openstack-infra06:34
*** luqas has joined #openstack-infra06:41
*** jcoufal has joined #openstack-infra06:43
* gus just had jenkins fail at "git remote update" - "Could not read from remote repository"06:46
AJaegergus, just recheck...06:55
AJaegerfungi, could you remove glance_store from jenkins so that the propose translation update works again, please? See
gusyep - I only did a brief bug search but couldn't see an existing e-r bug.  Since it seemed to hit many of my test runs at about the same time, I imagine others have seen it too.06:56
*** ildikov has joined #openstack-infra06:57
*** MaxV has joined #openstack-infra07:00
*** doude has joined #openstack-infra07:04
*** chandankumar has quit IRC07:07
*** MaxV has quit IRC07:09
*** ominakov has quit IRC07:09
*** chandan_kumar has joined #openstack-infra07:13
*** lcheng has quit IRC07:19
*** e0ne has joined #openstack-infra07:19
*** rushiagr has joined #openstack-infra07:20
rushiagrI was trying to create stories into storyboard, but it is not allowing me to07:21
rushiagrthe 'save changes' button is not clickable. Is this deliberately done?07:21
*** ashaeron has joined #openstack-infra07:21
rushiagrI found few issues with the storyboard webui, so was trying to report them07:22
ppaianteaya, could you relook at
*** flaper87 is now known as flaper87|afk07:26
ppaiSergeyLukjanov, could you relook at thanks :)07:26
*** Hal has joined #openstack-infra07:26
*** Hal is now known as Guest1045707:27
openstackgerritCarl Loa Odin proposed a change to openstack-infra/jenkins-job-builder: Extend trigger_builds functionality
openstackgerritA change was merged to openstack-infra/devstack-gate: Add oslo.utils and oslo.concurrency
*** shardy_afk is now known as shardy07:29
*** lyxus has quit IRC07:29
*** lyxus has joined #openstack-infra07:31
*** yfried_ has quit IRC07:32
*** harlowja_at_home has quit IRC07:33
*** chandan_kumar has quit IRC07:33
openstackgerritA change was merged to openstack/requirements: Remove docutils-pin
*** dizquierdo has joined #openstack-infra07:35
*** yfried has joined #openstack-infra07:35
vponomaryovfungi: just read thread, I am available for open question with moving Manila to openstack namespace.07:36
*** wendar_ is now known as wendar07:36
*** e0ne has quit IRC07:38
*** e0ne has joined #openstack-infra07:38
*** acruz has joined #openstack-infra07:38
*** flaper87|afk is now known as flaper8707:39
*** marun_ has quit IRC07:41
*** andreykurilin_ has quit IRC07:41
*** tsg has quit IRC07:42
*** jlibosva has joined #openstack-infra07:43
*** alexpilotti has joined #openstack-infra07:43
*** lcheng has joined #openstack-infra07:45
*** HeOS has joined #openstack-infra07:46
*** alexpilotti has quit IRC07:46
ttxlifeless: I replied to your thread. I'm not sure we actually have a problem here07:47
*** chandan_kumar has joined #openstack-infra07:47
*** mancdaz has joined #openstack-infra07:47
*** nuritv has joined #openstack-infra07:50
nuritvHi There, i have a Question - we have a CI system working on Neutron, and we want to create one for Cinder as well. do we need to register with a different account or can we use our existing account?07:52
openstackgerritA change was merged to openstack/requirements: Bump boto to 2.32.1
*** jistr has joined #openstack-infra07:53
*** jgallard has joined #openstack-infra07:53
*** alexpilotti has joined #openstack-infra07:54
*** MaxV has joined #openstack-infra07:55
*** imeerovi has joined #openstack-infra07:57
*** alexpilotti has quit IRC08:03
*** lcheng has quit IRC08:06
*** nuritv has quit IRC08:09
*** mattymo is now known as mmosesohn08:11
*** mmosesohn is now known as mattymo08:12
lifelessttx: ack08:14
*** nuritv has joined #openstack-infra08:15
*** sdake has quit IRC08:16
*** alexpilotti has joined #openstack-infra08:18
*** ZZelle has quit IRC08:18
*** ZZelle has joined #openstack-infra08:19
*** sdake has joined #openstack-infra08:20
*** melwitt1 has quit IRC08:24
*** derekh has joined #openstack-infra08:25
*** oomichi has quit IRC08:25
*** lttrl has quit IRC08:26
*** luqas has quit IRC08:28
*** chandan_kumar has quit IRC08:29
*** zz_johnthetubagu is now known as johnthetubaguy08:30
*** salv-orlando has joined #openstack-infra08:36
*** alexpilotti has quit IRC08:37
*** pblaho_ has joined #openstack-infra08:38
*** Longgeek has joined #openstack-infra08:39
*** andreykurilin_ has joined #openstack-infra08:42
*** koolhead17 has quit IRC08:42
*** lttrl has joined #openstack-infra08:44
*** ildikov_ has joined #openstack-infra08:44
*** tnovacik has joined #openstack-infra08:45
*** ildikov has quit IRC08:47
*** amuller has joined #openstack-infra08:50
*** mpaolino has joined #openstack-infra08:51
*** amotoki has quit IRC08:52
*** Longgeek has quit IRC08:53
*** nijaba has joined #openstack-infra08:53
*** Longgeek has joined #openstack-infra08:53
*** andreaf has joined #openstack-infra08:53
*** praneshp has quit IRC08:57
*** masayukig has quit IRC09:00
*** masayukig has joined #openstack-infra09:00
*** BobBall_Away is now known as BobBall09:01
*** jp_at_hp has joined #openstack-infra09:03
*** jp_at_hp has quit IRC09:03
*** jp_at_hp has joined #openstack-infra09:03
*** luqas has joined #openstack-infra09:04
*** nuritv has quit IRC09:05
*** andreykurilin_ has quit IRC09:06
*** jistr has quit IRC09:06
*** nuritv has joined #openstack-infra09:07
*** dtantsur|afk is now known as dtantsur09:07
*** yamamoto has quit IRC09:08
*** jistr has joined #openstack-infra09:10
*** andreykurilin_ has joined #openstack-infra09:10
*** andreykurilin_ has quit IRC09:15
*** andreykurilin_ has joined #openstack-infra09:18
*** alexpilotti has joined #openstack-infra09:21
openstackgerrityolanda.robla proposed a change to openstack-infra/config: Typo: replace puppet.rsh by puppet.rst
*** Longgeek has quit IRC09:24
*** alexpilotti has quit IRC09:26
*** Longgeek has joined #openstack-infra09:27
*** pblaho_ is now known as pblaho|afk09:32
*** e0ne has quit IRC09:35
*** marun_ has joined #openstack-infra09:39
*** andreykurilin_ has quit IRC09:39
*** ociuhandu has quit IRC09:40
*** tnovacik has quit IRC09:41
*** e0ne has joined #openstack-infra09:41
*** yamamoto has joined #openstack-infra09:42
*** nijaba has quit IRC09:46
*** nijaba has joined #openstack-infra09:46
*** amuller is now known as amuller_lunch09:49
*** zhiwei has joined #openstack-infra09:56
*** unicell has quit IRC10:00
*** _shaps__ is now known as _shaps_10:04
*** pcm_ has joined #openstack-infra10:05
*** pcm_ has quit IRC10:05
*** pcm_ has joined #openstack-infra10:06
*** jgallard has quit IRC10:08
*** yamahata has quit IRC10:11
*** nijaba has quit IRC10:12
*** dizquierdo has quit IRC10:16
*** pblaho|afk is now known as pblaho_10:20
*** koolhead17 has joined #openstack-infra10:20
*** saper has joined #openstack-infra10:22
*** e0ne has quit IRC10:24
openstackgerrityolanda.robla proposed a change to openstack-infra/config: Add pre-requisites needed for diskimage-builder
*** bhuvan has quit IRC10:25
*** imeerovi has quit IRC10:30
*** amuller_lunch is now known as amuller10:31
*** Longgeek has quit IRC10:31
*** Longgeek has joined #openstack-infra10:32
SergeyLukjanovanteaya, what's the correct specific recheck syntax for 3rd party?10:35
*** Longgeek_ has joined #openstack-infra10:36
*** saper has joined #openstack-infra10:36
*** masayukig has quit IRC10:39
*** Longgeek has quit IRC10:39
*** e0ne has joined #openstack-infra10:39
AJaegerFYI, last requirements post job failed due to tuskar-ui docutils pin,  a patch for tuskar-ui is merged, so next requirements post job should be fine...10:40
*** unicell has joined #openstack-infra10:40
*** naotok is now known as zz_naotok10:41
*** imeerovi has joined #openstack-infra10:42
*** ociuhandu has joined #openstack-infra10:42
*** yaguang has quit IRC10:44
*** daya_k has quit IRC10:46
*** Guest10457 has quit IRC10:46
*** pblaho_ is now known as pblaho10:49
*** _nadya_ has joined #openstack-infra10:49
*** bhuvan has joined #openstack-infra10:52
*** bhuvan has quit IRC10:56
*** jraim_ has quit IRC11:03
*** fungi has quit IRC11:03
*** comstud has quit IRC11:03
*** ttx has quit IRC11:03
*** MaxV has quit IRC11:04
*** miqui has quit IRC11:05
*** MaxV has joined #openstack-infra11:05
*** jraim__ has joined #openstack-infra11:06
*** dims has joined #openstack-infra11:06
*** dolphm has quit IRC11:06
*** ppai has quit IRC11:06
*** jyuso has quit IRC11:08
*** comstud has joined #openstack-infra11:08
*** ttx has joined #openstack-infra11:09
*** ttx has quit IRC11:09
*** ttx has joined #openstack-infra11:09
*** MaxV has quit IRC11:10
*** loki184 has quit IRC11:10
*** marun_ has quit IRC11:10
*** marun has joined #openstack-infra11:10
*** dolphm has joined #openstack-infra11:11
*** dims has quit IRC11:12
*** dims has joined #openstack-infra11:13
*** dims_ has joined #openstack-infra11:14
*** baoli has joined #openstack-infra11:16
*** fungi has joined #openstack-infra11:16
*** ppai has joined #openstack-infra11:17
*** dims has quit IRC11:17
krtaylorSergeyLukjanov, right now, "recheck" with or without anything after it, will restart jobs11:17
krtaylorSergeyLukjanov, sdague proposed a standard way of starting recheck for specific third-party systems, but it was abandoned11:18
krtaylorSergeyLukjanov, I plan on reopening that discussion soon11:19
*** pblaho is now known as pblaho|afk11:21
*** mpaolino has quit IRC11:21
krtaylorSergeyLukjanov, see
*** asettle has quit IRC11:28
*** asettle has joined #openstack-infra11:28
*** dims_ has quit IRC11:29
*** dims has joined #openstack-infra11:29
*** masayukig has joined #openstack-infra11:31
*** MaxV has joined #openstack-infra11:33
*** dims has quit IRC11:34
*** pelix has joined #openstack-infra11:36
*** dims has joined #openstack-infra11:37
*** mmedvede has joined #openstack-infra11:40
*** mmaglana has quit IRC11:41
*** mmaglana has joined #openstack-infra11:42
*** yamahata has joined #openstack-infra11:45
*** penguinRaider has joined #openstack-infra11:46
*** mmaglana has quit IRC11:47
*** masayukig has quit IRC11:48
*** masayukig has joined #openstack-infra11:48
*** flaviof_zzz is now known as flaviof11:49
*** dims has quit IRC11:50
*** dims has joined #openstack-infra11:50
*** Hal has joined #openstack-infra11:50
*** Hal is now known as Guest6435911:51
*** ppai has quit IRC11:51
*** dims_ has joined #openstack-infra11:51
*** Longgeek_ has quit IRC11:53
*** Longgeek has joined #openstack-infra11:54
*** dims has quit IRC11:54
*** Longgeek_ has joined #openstack-infra11:54
*** marun_ has joined #openstack-infra11:56
*** baohua_ has joined #openstack-infra11:58
*** yamahata has quit IRC11:58
*** Longgeek has quit IRC11:58
*** marun has quit IRC11:59
*** kgiusti has joined #openstack-infra12:00
*** yamahata has joined #openstack-infra12:00
*** KanagarajM has joined #openstack-infra12:00
*** baohua has quit IRC12:00
*** daya_k has joined #openstack-infra12:02
*** markmcclain has joined #openstack-infra12:02
*** markmcclain has quit IRC12:02
*** dprince has joined #openstack-infra12:03
*** erlon has joined #openstack-infra12:03
*** HenryG is now known as HenryG_afk12:04
*** markmcclain has joined #openstack-infra12:04
*** ppai has joined #openstack-infra12:04
*** mbacchi has joined #openstack-infra12:11
*** pblaho|afk is now known as pblaho12:12
*** mmaglana has joined #openstack-infra12:12
openstackgerritErno Kuvaja proposed a change to openstack/requirements: Limiting requests <2.4.0
*** mmaglana_ has joined #openstack-infra12:14
*** aysyd has joined #openstack-infra12:16
*** nosnos has quit IRC12:16
*** mmaglana has quit IRC12:17
*** nosnos has joined #openstack-infra12:17
*** Longgeek_ has quit IRC12:19
*** mmaglana_ has quit IRC12:19
*** nosnos has quit IRC12:21
*** adalbas has joined #openstack-infra12:21
*** dkranz has joined #openstack-infra12:21
*** miqui has joined #openstack-infra12:25
*** dims_ has quit IRC12:29
*** baohua_ has quit IRC12:30
*** dims has joined #openstack-infra12:30
*** Longgeek has joined #openstack-infra12:30
*** cnesa10 has quit IRC12:30
*** e0ne has quit IRC12:31
*** dims_ has joined #openstack-infra12:32
*** dims has quit IRC12:34
*** e0ne has joined #openstack-infra12:34
*** KanagarajM has quit IRC12:38
*** koolhead17 has quit IRC12:41
*** freyes has quit IRC12:45
*** dims_ has quit IRC12:45
*** freyes has joined #openstack-infra12:46
*** dims has joined #openstack-infra12:46
*** che-arne has quit IRC12:48
*** nijaba has joined #openstack-infra12:52
*** nijaba has joined #openstack-infra12:52
*** igor has joined #openstack-infra12:52
*** igor has quit IRC12:52
*** igor has joined #openstack-infra12:53
*** jgallard has joined #openstack-infra12:53
*** doug-fish has joined #openstack-infra12:53
openstackgerritKurt Taylor proposed a change to openstack-infra/config: An alternate recheck syntax for third-party CI systems
openstackgerritKiall Mac Innes proposed a change to openstack/requirements: Update oslosphinx to
*** marcusvrn has joined #openstack-infra12:55
*** igor has quit IRC12:56
openstackgerritKurt Taylor proposed a change to openstack-infra/config: An alternate recheck syntax for third-party CI systems
*** vponomaryov has quit IRC12:57
*** vponomaryov has joined #openstack-infra12:59
*** dkranz has quit IRC12:59
*** changbl has joined #openstack-infra13:00
*** sandywalsh has joined #openstack-infra13:01
*** enikanorov has joined #openstack-infra13:01
*** jpeeler has joined #openstack-infra13:05
*** zz_dimtruck is now known as dimtruck13:07
*** daya_k has quit IRC13:09
*** _nadya_ has quit IRC13:10
*** doude has quit IRC13:11
*** dkliban_afk is now known as dkliban13:12
*** doude has joined #openstack-infra13:12
*** mmaglana has joined #openstack-infra13:13
*** dkranz has joined #openstack-infra13:15
openstackgerritA change was merged to openstack-infra/git-review: Remove useless constants
*** enikanorov__ has joined #openstack-infra13:17
*** eharney has joined #openstack-infra13:18
*** mmaglana has quit IRC13:18
*** enikanorov has quit IRC13:19
*** mriedem has joined #openstack-infra13:21
*** radez_g0n3 is now known as radez13:22
*** craigbr has joined #openstack-infra13:24
*** julim has joined #openstack-infra13:24
*** dangers_away is now known as dangers13:25
*** bknudson has quit IRC13:26
*** mfer has joined #openstack-infra13:26
*** bookwar has joined #openstack-infra13:32
*** doug-fish1 has joined #openstack-infra13:36
*** mjturek has joined #openstack-infra13:36
*** jp_at_hp has quit IRC13:37
*** doug-fish has quit IRC13:39
*** ominakov has joined #openstack-infra13:41
*** e0ne has quit IRC13:43
*** bknudson has joined #openstack-infra13:44
*** ppai has quit IRC13:46
*** r-daneel has joined #openstack-infra13:47
*** e0ne has joined #openstack-infra13:50
*** ashaeron has quit IRC13:50
*** dimtruck is now known as zz_dimtruck13:53
*** homeless has joined #openstack-infra13:54
*** chuckC has quit IRC13:55
*** e0ne has quit IRC13:56
*** cdent has joined #openstack-infra13:56
*** dhellmann has quit IRC13:57
*** dhellmann has joined #openstack-infra13:58
*** e0ne has joined #openstack-infra13:58
fungiAJaeger: i cleared the workspaces on the proposal slave for glance_store. presumably a change landed to it after we renamed the project but before they merged my .gitreview update14:00
*** signed8bit has joined #openstack-infra14:02
*** pcrews has joined #openstack-infra14:04
fungivponomaryov: thanks, i was going to go venturing into #openstack-manila to figure out whether bswartz hangs out in irc. mainly we just want to figure out when a convenient time is to rename your program's git repositories from stackforge/.* to openstack/.*14:07
fungivponomaryov: if someone from the manila core team can be around next tuesday at 19:00 utc in #openstack-meeting (during the infra team meeting) to discuss scheduling that would be a huge help14:08
vponomaryovfungi: I can be there14:08
vponomaryovfungi: only time is open item?14:08
vponomaryovfungi: how long it takes? Whole process of moving?14:08
*** yfried has quit IRC14:09
*** zz_jgrimm is now known as jgrimm14:09
*** penguinRaider has quit IRC14:09
*** zz_dimtruck is now known as dimtruck14:10
openstackgerritA change was merged to openstack-dev/pbr: Look for and process sem-ver pseudo headers in git
*** wenlock has joined #openstack-infra14:13
*** nelsnelson has joined #openstack-infra14:14
*** annegent_ has joined #openstack-infra14:17
*** nuritv has quit IRC14:18
*** paulrad has joined #openstack-infra14:21
*** dmsimard_away is now known as dmsimard14:25
openstackgerritAndrey Nikitin proposed a change to openstack-infra/jeepyb: Update mechanism of creating groups in database and managing access rights with jeepyb
marcusvrnGood morning, just a the CI system page, under requirements section, there's a topic (All comments from your CI system must contain a link to the wiki page for your CI system.). This topic means that the CI system have to comment on patches....but the CI system doesn't have that behavior (comment on patches) anymore. What should I do?14:26
*** otherwiseguy has joined #openstack-infra14:26
zaroanteaya: i don't think you can restrict a gerrit user from adding comments to a change.  you can only restrict them from voting14:26
fungivponomaryov: yeah, basically we just need to know when will be convenient for your project, and then we coordinate available time among infra root sysadmins to determine when a couple of us will be available to make the change (since it involves a brief outage for zuul and gerrit, and some manual steps we have to perform on the servers)14:27
*** tonytan4ever has joined #openstack-infra14:27
vponomaryovfungi: I guess we can just set it for more convenient time and inform all folks in manila room.14:28
*** rushiagr is now known as rushiagr_away14:28
zaromarcusvrn: link to wiki?14:28
fungivponomaryov: if all we're doing is renaming those two projects, then it will take us probably roughly half an hour. we usually like to do them on fridays or weekends when the developer activity is at a minimum14:28
vponomaryovfungi: convenient time for you14:28
*** andreaf has quit IRC14:28
zaromarcusvrn: where's the ci system page?14:29
zaroanteaya: ^14:30
openstackgerritJames E. Blair proposed a change to openstack-dev/pbr: Accept capitalized Sem-Ver headers
vponomaryovfungi: or send ML with "manila" tag, it is even better, but time choosing is up to you14:31
*** wenlock has quit IRC14:31
openstackgerritAndrey Nikitin proposed a change to openstack-infra/gerritlib: Update listGroups function in
fungivponomaryov: okay, thanks! we'll pick a time and then post to the -dev ml when we're ready to schedule it14:31
zaromarcusvrn: are you not the owner of that CI system?14:32
marcusvrnzaro: where can I link to wiki? the comments are no more shown.. :(14:32
vponomaryovfungi: ok14:32
*** jcoufal_ has joined #openstack-infra14:32
marcusvrnzaro: nope, I only have the credentials14:33
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Update pep8 ignore list
*** jcoufal has quit IRC14:34
*** mmaglana has joined #openstack-infra14:35
zaromarcusvrn: can you maybe ask the owner to comply with that requirement?14:35
*** HenryG_afk is now known as HenryG14:36
*** ildikov has joined #openstack-infra14:36
*** david-lyle has joined #openstack-infra14:36
*** eharney_ has joined #openstack-infra14:36
*** stevemar has joined #openstack-infra14:36
openstackgerritJames E. Blair proposed a change to openstack-dev/pbr: Accept capitalized Sem-Ver headers
*** eharney has quit IRC14:37
*** eharney_ is now known as eharney14:37
*** chuckC has joined #openstack-infra14:38
*** gargola has joined #openstack-infra14:38
*** cody-somerville has joined #openstack-infra14:39
*** bswartz has joined #openstack-infra14:39
jeblairfungi: shall we restart nodepool?14:40
*** ildikov_ has quit IRC14:40
marcusvrnzaro: sry, I misunderstood your question....I'm deploying the CI, so I have to do it14:40
fungijeblair: let's do this thing. no time like the present14:41
fungishall i try to gracefullt stop it?14:41
fungier, gracefully14:41
*** ildikov has quit IRC14:41
jeblairfungi: does that work?  i thought it didn't.  or at least, not well.14:42
fungiit used to...14:42
jeblairfungi: i guess we can try it for a few minutes :)14:42
fungijust did, stopped instantly14:42
fungifrom what i've seen it either works immediately or not at all14:42
fungishall i start it back up now?14:43
fungithere it goes, deleting building nodes14:43
fungilaunching new servers14:43
fungiOverLimit: Quota exceeded for ram: Requested 8192, but already used 761856 of 7614:44
fungi8000 ram14:44
*** yfried has joined #openstack-infra14:44
jeblairfungi: we might want to run an alien check and also look for used > a few hours in a few hours.14:44
fungii will add that to my morning plans14:44
*** zz_gondoi is now known as gondoi14:45
dolphmcan someone clarify if we should be using reverify at all anymore, or if it's *only* "recheck <reasoning>" now?14:46
AJaegerfungi, the gitreview only landed last night and the proposal job run before. Thanks!14:46
openstackgerritSylvain Bauza proposed a change to openstack-infra/config: Enable IRC channel logging for #openstack-fr
*** datsun180b has joined #openstack-infra14:46
fungidolphm: they're synonymous now14:47
marun_jogo: re:
dolphmfungi: thank you! cc- ttx14:47
*** atiwari has joined #openstack-infra14:47
fungidolphm: more or less have been ever since we added the clean check requirement, since you couldn't reenqueue into the gate after failing out without rerunning check jobs. but about a month ago we just went ahead and made them completely synonymous (about the same time we stopped requiring a bug number or a "no bug")14:48
marun_jogo: given this following traceback, can you help me determine a more appropriate elasticsearch query?14:48
marun_jogo: The problem can apparently reproduce on other tests that use eventlet, btw, though the l3 agent test is the only cause in the gate.14:49
AJaegerfungi, I suggest that the .gitreview gets approved as part of the migration - it will cause problems otherwise...14:49
*** amuller_ has joined #openstack-infra14:49
*** koolhead17 has joined #openstack-infra14:49
*** koolhead17 has joined #openstack-infra14:49
*** unicell has quit IRC14:50
fungiAJaeger: we may want to rethink how that's being used to add gerrit remotes in jobs, so that it can stay decoupled14:50
dolphmfungi: good to know, thanks14:50
AJaegerfungi, that's another option...14:51
*** tsg has joined #openstack-infra14:51
*** amuller has quit IRC14:51
fungijeblair: so far so good... not jumping significantly high on node count yet, at least14:52
*** wenlock has joined #openstack-infra14:53
zhiyanfungi: may i get your 2 mins? met a probably in job,
fungibuilding count is falling and used count increasing, though it's still trying to worth through deletions from earlier i think14:54
zhiyanfungi: i have no idea if this is an known issue?14:54
*** jcoufal_ has quit IRC14:55
fungizhiyan: did you look in the devstack logs?14:55
fungizhiyan: "Running devstack ... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz) ... ERROR: the main setup script run by this job failed - exit code: 1 ... please look at the relevant log files to determine the root cause"14:56
zhiyani could met that issue in as's a rush fix to unblock some changes..14:56
jeblairfungi: a lot of deletion timeouts and a lot of 'ERROR' states from rax14:57
fungizhiyan: looks like someone might have broken glanceclient14:57
fungizhiyan: oh, it's you14:57
fungizhiyan: your change breaks glanceclient14:57
zhiyanfungi: oh? which one?14:58
fungizhiyan: 11862714:58
*** bhuvan has joined #openstack-infra14:58
fungizhiyan: please look at the test logs, or test your change yourself. integration tests appear to be doing precisely what they're designed to do--keeping your change from merging and breaking glanceclient14:59
zhiyanfungi: actually we just want to use 118627 to fix the issue which latest requests provided14:59
*** pblaho is now known as pblaho|afk14:59
*** rushiagr_away is now known as rushiagr14:59
zhiyanfungi: without that change the glanceclient will be failed, actually...14:59
*** mriedem has quit IRC15:00
*** mriedem has joined #openstack-infra15:01
fungizhiyan: with that change it fails in a different way ("from requests.packages import urllib3 == ImportError: No module named packages")15:01
zhiyanfungi: because the version which jenkins installed is not the latest version..15:01
fungizhiyan: the version of glanceclient being used in that job is missing the change being tested?15:03
fungizhiyan: that doesn't seem to be the case at all15:03
fungizhiyan: the line which is throwing an exception is a line added by your proposed change15:04
zhiyanfungi: now i'm not sure why, but from a older test result, i can see a new exception (ProtocolError) which only latest requests provided (2.4.0) could be triggered, so that's why we prepared #118627..15:04
zhiyanfungi: yes, , the reason.15:04
fungizhiyan: sure, but you can't "from requests.packages import urllib3" because requests.packages is not a module15:04
uvirtbotLaunchpad bug 1364893 in python-glanceclient "New version of requests library breaks unit tests" [Undecided,In progress]15:04
fungizhiyan: that's what the python excpetion being raised means15:05
zhiyanfungi: ok, will try. thanks.15:06
*** zhiwei has quit IRC15:07
*** pblaho|afk is now known as pblaho15:07
*** _nadya_ has joined #openstack-infra15:08
fungizhiyan: that job ran with requests==2.2.115:09
fungizhiyan: according to
zhiyanfungi: interesting..but recently i see it 2.4.015:09
*** dizquierdo has joined #openstack-infra15:10
fungizhiyan: something else being installed in that integration test has that pinned, so it's running with an earlier version than 2.4.015:12
*** imeerovi has quit IRC15:12
*** mmaglana has quit IRC15:12
zhiyanfungi: hum..for this issue, i don't think pining is a good idea, i think we'dbetter just fix it15:13
*** bdpayne has joined #openstack-infra15:13
*** mmaglana has joined #openstack-infra15:13
zhiyanfungi: but back to this topic, i see global requment didn't pin it.
zhiyanfungi: so do you think we could just skip 118627? (due to we will not use 2.4.0 , and gate will be unblocked), but i think it's not a stable solution....15:15
fungizhiyan: right, i don't think one of our projects is likely pinning it... some transitive dependency probably is15:15
jrollmight be better to pin global-reqs at < 2.4.015:15
*** AJaeger has quit IRC15:15
fungizhiyan: no, i think it needs to be fixed, but integration testing is complicated when you're trying to interoperate with other dependencies which may declare dependencies on different versions of things you use15:16
jroll(I think we should be pinning at == $version for everything, but that's a different conversation)15:16
*** gokrokve has joined #openstack-infra15:16
zhiyanfungi: do you think pining <2.4.0 is a idea?15:16
fungijroll: that works until you start depending on things which depend on things15:16
*** Guest64359 has quit IRC15:17
fungijroll: and also if you don't care whether the things you depend on have security holes fixed in newer releases15:17
*** mmaglana has quit IRC15:17
*** rlandy has quit IRC15:17
*** rhe00 has left #openstack-infra15:18
*** Hal has joined #openstack-infra15:18
jrollfungi: I realize the problem is much harder than me waving my hands around :)15:18
*** Hal is now known as Guest4228615:18
fungijroll: i frequently run around with flailing kermit arms15:19
*** vhoward has left #openstack-infra15:19
*** doug-fish1 has left #openstack-infra15:20
fungizhiyan: aha! so the reason it's using requests==2.2.1 is that we apt-get install python-requests on ubuntu trusty for devstack15:20
*** acruz has quit IRC15:21
zhiyanfungi: can you help fix it?15:21
jrollI'd be willing to bet it's a dependency for some other package installed by apt15:22
fungizhiyan: with the python-requests deb installed, i can recreate this error. not sure if it's a modification of the requests library by the debian package maintainer or something else odd15:22
sdaguezhiyan: glance client should probably be able to bridge across this15:22
*** jcoufal has joined #openstack-infra15:22
jrollthat's a *lot* of python packages installed by apt :(15:22
*** pblaho is now known as pblaho|afk15:22
zhiyansdague: it's a temporary way can go, but good enough15:23
*** paulrad has quit IRC15:23
*** liam__ has joined #openstack-infra15:23
*** hdd has joined #openstack-infra15:23
*** acruz has joined #openstack-infra15:23
*** paulrad has joined #openstack-infra15:24
zhiyanfungi: jroll yep, i see the point15:24
*** torandu has left #openstack-infra15:24
*** ominakov has quit IRC15:24
sdagueso python-boto is what's dragging it in on devstack15:26
sdaguewhich, honestly, removing from the package list wouldn't be a terrible idea15:26
*** Sukhdev has joined #openstack-infra15:26
fungizhiyan: yep, so it's a local change introduced into debian's python-requests package...
zhiyani saw novaclient has this same issue as well, probably needs bridge as well in its code15:27
*** pballand has joined #openstack-infra15:27
*** ominakov_ has joined #openstack-infra15:27
*** paulrad has quit IRC15:28
fungiit appears the workaround is to try: from requests.packages import urllib3 and then except ImportError: import urllib315:28
zhiyanok, thanks fungi sdague inputs.15:28
*** acruz is now known as arxcruz15:28
fungisince python-urllib3 is packaged separately15:29
*** ominakov_ has quit IRC15:29
openstackgerritDmitry Teselkin proposed a change to openstack-infra/config: Pin pupppetdb-terminus package
fungijeblair: hrm... we have a glut of ready nodes now. checking for any obvious imbalance/staleness15:30
jeblairyeah i was just noticing that15:30
jeblairfungi: they are all very new15:30
jeblairoh nm, not that new15:30
*** liam__ has quit IRC15:30
jeblairsome of them are several 10s of minutes old15:31
fungithat may be some sort of hysteresis and then lag in going from nodepool-ready to jenkins-ready15:31
*** Sincler has joined #openstack-infra15:32
fungithe bulk are in rax15:32
fungibut fairly rapidly falling15:32
jeblairalmost no one is waiting on a centos job :/15:33
*** amotoki has joined #openstack-infra15:34
*** alexpilotti has joined #openstack-infra15:34
*** pblaho|afk is now known as pblaho15:34
*** skolekonov has quit IRC15:34
fungiswitching to an overlap graph instead of a stack graph makes it look like we had an influx of ready as the building sharply fell, and then a lot of delete as the ready fell, so presumably just nodepool managing to get well in front of the demand?15:35
jeblairfungi: well in front of the demand isn't really desirable when we're at capacity15:36
fungibuilding is climbing again as well15:36
jeblairi mean, nodes sitting for 30 minuts doing nothing isn't great15:36
YorikSarmordred: ping15:36
fungitrying to figure out whether this is similar to what we saw the last time, minus the jumping waaay above quota behavior15:36
*** gokrokve has quit IRC15:37
YorikSarmordred: As I understand, it was you who switched git-review to bare man page instead of Sphinx-generated one.15:37
YorikSarmordred: Maybe we can switch back and use wheels instead?15:37
fungiit looks like building jumps in 10-minute intervals, then falls sharply, then starts to climb again15:37
jeblairfungi: oh, most of them are precise15:37
jeblairfungi: i get it15:37
*** jgallard has quit IRC15:37
clarkbI take it nodepool was restarted /me reads sb15:37
fungiclarkb: yup15:37
jeblairfungi: the round robin allocator is satisfying all of the "demand" for devstack-precise nodes when it gets to be their turn15:38
jeblairfungi: the "demand" for them is from min-ready15:38
fungiso we jump by as much as min-ready for that type15:38
jeblairfungi: normally, min-ready has little impact when at capacity15:38
jeblairbut now we're giving it outsized importance15:38
*** gokrokve has joined #openstack-infra15:39
clarkboh right, we used it to compensate before but now we are getting it all15:39
*** annegent_ has quit IRC15:39
*** yamahata has quit IRC15:39
*** liam__ has joined #openstack-infra15:39
*** andreaf has joined #openstack-infra15:39
*** johnthetubaguy is now known as zz_johnthetubagu15:40
mordredYorikSar: why do man pages have anything to do with wheels?15:40
YorikSarmordred: We can package them in wheel and user who installs this wheel won't need Sphinx to get manpage.15:41
*** zz_johnthetubagu is now known as johnthetubaguy15:41
mordredwell, kinda. except that we don't know where to install the man page until install time15:42
mordredbecause, well, insanity15:42
*** andreaf_ has joined #openstack-infra15:42
fungimordred: YorikSar: also we don't currently have a way to specify different requirements for tarballs vs wheels, right?15:43
mordredso, we can make the manpage  generated by Sphinx, but it will not solve or change anything15:43
*** mflobo has quit IRC15:43
mordredfungi there is no way to do that, that's right15:43
*** liam__ has quit IRC15:43
YorikSarmordred: Hm... How does people package manpages then?..15:44
*** zeus has quit IRC15:44
YorikSarfungi: I think setup_requires should do that.15:45
*** chuckC has quit IRC15:45
*** zeus has joined #openstack-infra15:45
YorikSarfungi: With wheels don't get executed at all.15:45
*** paulrad has joined #openstack-infra15:45
*** andreaf has quit IRC15:45
fungiYorikSar: usually developers package manpages as static files (or often not at all). also manpages via pip install are sort of broken due to it being hard to maintain cross-platform support for all the various places they could go (and in a virtualenv they make basically no sense whatsoever)15:46
openstackgerritA change was merged to openstack-infra/subunit2sql: Update pep8 ignore list
fungion a related note, based on a recent thread on debian-devel i'm getting the feeling that a lot of distro packagers now view upstream tarballs as at best a relic, at worst a nuisance, and their use in distribution source packages a mere implementation detail15:48
fungiseems many would rather upstreams just tagged a commit in vcs and left instructions on how to deploy that so it can be translated into packaging15:48
*** emagana has joined #openstack-infra15:49
mordredI'm fine with that as long as they don't argue with me on build tooling :)15:49
mordredthat is, our releases are all tagged and content is 100% reproducible15:49
YorikSarfungi: That's interesting. I wonder if we ever give up tarballs :)15:49
mordredYorikSar: setuptools is especially dumb wrt manpages15:50
fungiin which case, if we accept this as a commonplace attitude, we could just start considering our tarballs to purely be the pypi packaging mechanism rather than trying to make it double as "upstream source"15:50
jeblairmordred: i can't imagine anyone having a discussion on build tooling that isn't an argument :)15:50
mordredjeblair: :)15:50
mordredfungi I'm in large favor of admitting that our source is in git15:50
mordredsince it is15:50
*** mmaglana has joined #openstack-infra15:51
YorikSarmordred: All setuptools installed manpages to to ~/.local/share/man on my machine and that's what I've added to MANPATH :)15:51
mordredyah. systemwide it gets brokener15:52
pleia2title could have been slightly better, but hey hey
fungiYorikSar: does that work if you install into a virtualenv? i've not had a lot of luck getting that to pan out15:52
*** trinaths1 has joined #openstack-infra15:52
mordredthe problem isn't that15:53
pleia2mordred: welcome back, I made a stick puppet you for puppetconf
mordredit's that manpaths are different on different systems15:53
*** trinaths1 has left #openstack-infra15:53
*** chuckC has joined #openstack-infra15:53
YorikSarfungi: You can probably use relative paths in MANPATH (like ./.tox/py27/share/man), but I have never wanted that.15:53
mordredpleia2: I saw! thank you!!15:53
*** emagana has quit IRC15:53
pleia2mordred: yay :)15:53
openstackgerritDolph Mathews proposed a change to openstack-infra/elastic-recheck: Add query for critical keystone regression
mordredYorikSar: you CAN do anything, but we needs to be able to install properly into default systems15:54
*** emagana has joined #openstack-infra15:55
YorikSarmordred: Sure. I'm just saying that using pip/setuptools to install things into otherwise-package-controlled system won't generally work anyway.15:55
dolphmjogo: mtreinish: sdague: elastic-recheck query for a critical keystone regression introduced about 2 hours ago:
mordredbut it does work noe15:55
mordredand I'm not going to break that without good reason15:56
*** arxcruz has quit IRC15:56
mordredthe "don't mix pip and apt" mantra is unrealistic ... it regularly happens15:56
*** bdpayne has quit IRC15:57
dolphmsdague: thank you15:57
sdaguemordred: I've started some selective pruning of apt installed python that's pure python, as I don't think we gain anything from it in devstack.15:57
sdaguebut for python/c stuff, I think we need to stay with apt15:57
mordredsdague: I think we should install nothing from apt that we can install from pip15:58
mordredfor consistency15:58
YorikSarmordred: Ok, now I'm confused. How does it work now and why won't it work the same way with wheel?..15:58
sdaguemordred: so .... gentoo?15:58
*** dtantsur is now known as dtantsur|afk15:58
YorikSarsdague: Yay! Gentoo!15:58
mordredYorikSar: right now we run code in to figure out the right path at install time15:58
sdagueI really don't want to be compiling source for 30 minutes on a new devstack15:59
mordredsince wheel doesn't execute that, wheel git-review will put manpages in the wrong place on some platforms15:59
clarkbsdague: we can probably mitigate a lot of that with ccache and pip installs to a tmp virtualenv in our image build process15:59
*** bdpayne has joined #openstack-infra15:59
mordredsdague: we have other solutions for tjat15:59
YorikSarmordred: Who decides where to put them? pbr?15:59
mordredYorikSar: yes15:59
mordredsdague: Ubuntu is fine ... apt is fine ...16:00
YorikSarsdague: We can build binary package repo for everythin but our stuff :)16:00
mordredsdague: but if it's in global reqs, we should install from pip16:00
YorikSarmordred: Ah... Magic... Ok...16:00
sdaguemordred: there are other gate solutions16:00
*** enikanorov has joined #openstack-infra16:00
sdaguebut not other solutions for developers getting started16:01
mordredsdague: there are other dev solutions too16:01
mordredyes there are16:01
mordredbut I'm on my phone ... :)16:01
sdaguewell, propose away. I mostly just don't want to turn the 10 minute devstack install time into an hour long process, especially when much of that is available as prebuilt binaries16:01
*** Longgeek has quit IRC16:02
YorikSarsdague: We can prebuild distro-specific wheels for gate.16:02
sdagueYorikSar: I'm not really talking about the gate here16:02
sdaguethat's a different optimization16:03
YorikSarsdague: Ah...16:03
*** sweston_ has joined #openstack-infra16:03
sdaguedevstack is actually used by people outside of the gate :)16:03
clarkbfungi: Bug 1365046 is weird and potentially an incidence of that rax thing? but this time it is in ord16:03
uvirtbotLaunchpad bug 1365046 in openstack-ci "Job failed due to no devstack directory" [Undecided,New]
*** TravT has joined #openstack-infra16:03
mordredI actually don't really care. but we have global requirements, and we subvert that in the gate for a random collection of things. in most other cases, we try for correctness and then fix the problem16:03
*** marcoemorais has joined #openstack-infra16:03
YorikSarOutside CI devstack tends to take way more than 10 minutes for me.16:03
*** cbkyeoh has joined #openstack-infra16:03
jeblairYorikSar: we cache most of what it needs on the node already, so there's very little network traffic16:04
*** eglynn__ has joined #openstack-infra16:04
*** vish1 has joined #openstack-infra16:04
mordredso, we can keep arbitrarily installing some python things from Ubuntu and I'm fine with it ... but it is surprising given g-r16:04
*** pelix1 has joined #openstack-infra16:04
*** hogepodge has joined #openstack-infra16:04
TravTok,probably a dumb question.  will zuul attempt an auto rebase if a dependent patch is not based on the most recent parent patch?16:05
clarkbhrm does show openstack-dev/devstack getting updated which is probably the cause?16:05
*** bmwiedemann1 has joined #openstack-infra16:05
*** primemin1sterp has joined #openstack-infra16:05
*** markvan_ has joined #openstack-infra16:06
*** bswartz1 has joined #openstack-infra16:06
*** willroberts_ has joined #openstack-infra16:06
*** simonmcc____ has joined #openstack-infra16:06
clarkb that will do it16:06
*** tristanC_ has joined #openstack-infra16:06
clarkbfungi: false alarm not related to that imge thing16:06
sdaguemordred: well lets deal with the easy parts first, where I don't think there is any drawback, then deal with the other parts later16:07
*** afazekas has quit IRC16:07
*** thingee_ has joined #openstack-infra16:07
*** Adri2000_ has joined #openstack-infra16:07
sdagueinstead of start at the hard parts, get in a fight, and never do the easy parts :)16:07
*** NobodyCa1 has joined #openstack-infra16:07
*** masayukig_ has joined #openstack-infra16:07
*** dosaboy_ has joined #openstack-infra16:07
fungiclarkb: it hit some sort of weird https issue cloning nova, and the setup-workspace script exited nonzero as a result16:07
*** Adri2000_ is now known as Guest6090916:07
*** doude_ has joined #openstack-infra16:08
clarkbfungi: ya16:08
*** ctracey_ has joined #openstack-infra16:08
sdaguespeaking of which - - which would have exposed the requests thing more places16:08
*** signed8b_ has joined #openstack-infra16:08
*** ctracey has quit IRC16:08
*** sweston has quit IRC16:08
*** vishy has quit IRC16:08
*** vish1 is now known as vishy16:08
*** cyeoh has quit IRC16:08
*** pelix has quit IRC16:08
*** SumitNaiksatam has quit IRC16:08
*** primeministerp has quit IRC16:08
*** skath_ has quit IRC16:08
*** Mithrandir has quit IRC16:08
*** erw has quit IRC16:08
*** msuriar has quit IRC16:08
*** k4n0 has quit IRC16:08
*** bswartz has quit IRC16:08
*** sileht has quit IRC16:08
*** rossella_s has quit IRC16:08
*** eglynn has quit IRC16:08
*** AaronGreen has quit IRC16:08
*** tristanC has quit IRC16:08
*** dougwig has quit IRC16:08
*** enikanorov_ has quit IRC16:08
*** freyes has quit IRC16:08
*** tchaypo has quit IRC16:08
*** StevenK has quit IRC16:08
*** rushiagr has quit IRC16:08
*** luqas has quit IRC16:08
*** mancdaz has quit IRC16:08
*** simonmcc___ has quit IRC16:08
*** maurosr has quit IRC16:08
*** signed8bit has quit IRC16:08
*** Svedrin has quit IRC16:08
*** NobodyCam has quit IRC16:08
*** markvan has quit IRC16:08
*** thingee has quit IRC16:08
*** dosaboy has quit IRC16:08
*** masayukig has quit IRC16:08
*** Adri2000 has quit IRC16:08
*** krtaylor has quit IRC16:08
*** GheRivero has quit IRC16:08
*** doude has quit IRC16:08
*** johnthetubaguy has quit IRC16:08
*** asadoughi has quit IRC16:08
*** willroberts has quit IRC16:08
*** mancdaz has joined #openstack-infra16:08
*** cbkyeoh is now known as cyeoh16:08
*** Mithrandir_ has joined #openstack-infra16:08
*** ctracey_ is now known as ctracey16:08
*** sweston_ is now known as sweston16:08
*** msuriar has joined #openstack-infra16:08
*** tristanC_ is now known as tristanC16:08
*** bswartz1 has quit IRC16:08
*** tchaypo has joined #openstack-infra16:09
*** luqas has joined #openstack-infra16:09
*** rossella_s has joined #openstack-infra16:09
*** k4n0_ has joined #openstack-infra16:09
*** bswartz has joined #openstack-infra16:09
fungiclarkb: which begs the question why it tried to go ahead with the gate hook16:09
sdaguemordred: also... aren't you supposed to be on a beach ?16:09
*** johnthetubaguy has joined #openstack-infra16:09
clarkbfungi: right, it should short circuit hard there imo16:09
*** maurosr has joined #openstack-infra16:09
*** skath_ has joined #openstack-infra16:09
*** asadoughi has joined #openstack-infra16:09
*** erw_ has joined #openstack-infra16:09
*** rushiagr has joined #openstack-infra16:10
*** dougwig_ has joined #openstack-infra16:10
*** krtaylor has joined #openstack-infra16:10
*** Svedrin has joined #openstack-infra16:10
*** freyes has joined #openstack-infra16:10
*** AaronGr has joined #openstack-infra16:10
*** amcrn has joined #openstack-infra16:10
*** sileht has joined #openstack-infra16:10
*** sileht has quit IRC16:10
*** che-arne has joined #openstack-infra16:10
*** bswartz has quit IRC16:10
*** sileht has joined #openstack-infra16:10
jeblairfungi: the ready spread doesn't seem so egregious now16:11
*** bswartz has joined #openstack-infra16:11
jeblairit looks like it's a bit more than before, but not radically so16:11
*** trinaths has joined #openstack-infra16:11
*** thingee_ is now known as thingee16:11
clarkbjeblair: do we need to udpate our min ready values to be a lot more conservative? since we should expand to meet demand better now?16:11
*** krtaylor has quit IRC16:11
fungijeblair: right--at this point i;m mostly evaluating it by how/whether it keeps up with the check pipeline backlog while still fulfilling all gate pipeline needs in a timely fashion16:11
*** trinaths has left #openstack-infra16:12
fungiclarkb: i think that robs us of faster job results during quiet periods if we do16:12
jeblairclarkb: i think min-ready is so that .. yeah what fungi said16:12
*** simonmcc____ is now known as simonmcc___16:12
*** pblaho has quit IRC16:12
jeblairfungi: i noticed lower gate was lagging on devstack-trusty a while ago, but looks ok now16:13
*** StevenK has joined #openstack-infra16:13
*** GheRivero has joined #openstack-infra16:13
fungiyeah, briefly16:13
*** bhuvan has quit IRC16:13
clarkbhrm good point, it is there to combat lag when we are quieter16:14
fungithough at the expense of making the logic a bit more complicated, we could scale min-ready downward the more we exceed it in use16:14
mordredsdague: I am. I'm not here16:14
fungibut that seems like it would get unnecessarily fiddly16:15
jeblairi notice a lot of red in the gate due to socket timeouts to pypi.o.o16:15
jeblairi think hpcloud<->rax may have had a network thing16:15
*** habib has joined #openstack-infra16:16
*** willroberts_ is now known as willroberts16:16
clarkbjeblair: that may be related to the TLS thing I linked to as well? though that was ord to dfw16:16
*** andreaf_ is now known as andreaf16:17
jeblairclarkb: oh i missed your followup on that.  hrm.16:17
*** paulrad has quit IRC16:17
*** paulrad has joined #openstack-infra16:17
*** bdpayne_ has joined #openstack-infra16:17
jeblairclarkb: well, there's an 8 hour time difference16:18
clarkboh ya16:18
*** MaxV has quit IRC16:18
*** MaxV has joined #openstack-infra16:19
*** paulrad has quit IRC16:19
*** paulrad_ has joined #openstack-infra16:19
*** gyee has joined #openstack-infra16:20
*** bdpayne has quit IRC16:21
*** amuller__ has joined #openstack-infra16:22
*** mwagner_lap has quit IRC16:22
*** MaxV has quit IRC16:23
*** NobodyCa1 is now known as NobodyCam16:24
*** paulrad_ has quit IRC16:24
*** annegent_ has joined #openstack-infra16:25
*** amuller_ has quit IRC16:26
clarkbthe zuul events queue is fairly large16:28
*** cody-somerville has quit IRC16:29
*** annegent_ has quit IRC16:30
jeblairnow it's small16:31
*** cody-somerville has joined #openstack-infra16:31
*** Guest42286 has quit IRC16:32
clarkbthat was quick. I guess the event queue processing backs it up occasionally16:32
jeblairresult queue processing is generally slower; it causes queue walks16:32
jeblairbut it has priority over event queue processing16:32
clarkbright because results can affect how we process events16:33
jeblairclarkb: most of the backloge in the trigger event queue was pseudo events for a merged change16:34
jeblairso in that case, we have hundreds of trigger events appear all at once, and they get processed pretty quickly too16:34
*** bhuvan has joined #openstack-infra16:35
*** koolhead_ has joined #openstack-infra16:35
clarkbI see16:35
*** jistr has quit IRC16:35
*** _nadya_ has quit IRC16:36
*** terryw has joined #openstack-infra16:37
*** krtaylor has joined #openstack-infra16:37
jeblairwow, neutron has some very old open changes16:37
*** otherwiseguy has quit IRC16:37
*** koolhead17 has quit IRC16:37
*** praneshp has joined #openstack-infra16:37
*** luqas has quit IRC16:37
*** annegent_ has joined #openstack-infra16:38
*** jlibosva has quit IRC16:38
clarkbjeblair: some additional changes just merged and the event queue did grow quickly16:39
clarkbjeblair: yay meta events :) ?16:39
*** dizquierdo has quit IRC16:40
*** mriedem1 has joined #openstack-infra16:40
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Fix method name call when files cannot be found
*** ildikov has joined #openstack-infra16:41
jeblairclarkb: yeah, pushes our numbers up without doing much more work! :)16:42
*** annegent_ has quit IRC16:43
fungilooks like we're currently running ~2hr wait for devstack-trusty nodes in the check pipeline16:43
*** dkranz has quit IRC16:43
*** mriedem has quit IRC16:44
*** bdpayne_ has quit IRC16:44
*** hogepodge has quit IRC16:44
clarkbfungi: do we know what the wait was before the nodepool restart when under load?16:44
*** arnaud has joined #openstack-infra16:45
fungiclarkb: it was probably about the same at peak yesterday. don't really have a good statistical sample to go on though16:45
fungimainly waiting to see whether it gets substantially worse once we've been running with the new algorithm for a few hours16:46
clarkbyou guys won't throw produce at me if I suggest another node.js service as a potential alternative to znc as a service will you?16:46
clarkbI guess I can respond to the thread and see how many tomatos you have16:47
fungii want to turn it into a service-as-a-service meta-discussion16:48
*** hdd has quit IRC16:49
mordredclarkb: node.js is a thing. we might as well admit it16:50
clarkbmordred: there is a difference between storyboard using it to build and test and running production applications on it16:51
clarkbmordred: but I agree, I don't expect it to go anywhere and it is pretty damn popular16:51
*** hogepodge has joined #openstack-infra16:51
mordredavoiding it isn't helpful generally16:52
jeblairclarkb: i'm pretty sure it's already dead to the hacker news crowd16:52
fungii want a sandboxed interpreter for basic, and a package manager for my basic applications16:52
*** doude_ has quit IRC16:52
morganfainbergmordred, welcome back btw.16:52
*** andreykurilin_ has joined #openstack-infra16:52
mordredmorganfainberg: not really back ... I'm at the beach phoning over breakfast :)16:53
mordredmorganfainberg: but thanks!16:53
jeblairmorganfainberg: this may be the new normal for mordred16:53
morganfainbergjeblair, hehe16:53
morganfainbergmordred, beach, breakfast, gee, what an awful way to spend a wednesday :)16:54
clarkbis it wednesday already16:54
morganfainbergmordred, well enjoy the rest of the "not really here" time :)16:55
*** paulrad has joined #openstack-infra16:56
fungiclarkb: lies16:56
*** alkari has joined #openstack-infra16:58
clarkbpretty sure it is only tuesday16:58
*** johnthetubaguy is now known as zz_johnthetubagu16:58
clarkbmonday was a time warp16:58
fungioxymoronic labor holiday is oxymoronic16:59
*** emagana has quit IRC16:59
*** emagana has joined #openstack-infra16:59
*** tsg has quit IRC17:00
*** marun_ has quit IRC17:00
*** dougwig_ is now known as dougwig17:00
*** tonytan4ever has quit IRC17:00
*** marcoemorais has quit IRC17:00
*** derekh has quit IRC17:01
*** paulrad has quit IRC17:01
*** tsg has joined #openstack-infra17:02
*** marcoemorais1 has joined #openstack-infra17:02
*** amcrn_ has joined #openstack-infra17:02
*** gema has quit IRC17:02
*** jcoufal has quit IRC17:03
*** gokrokve_ has joined #openstack-infra17:03
*** emagana has quit IRC17:04
*** tgohad has joined #openstack-infra17:04
*** emagana has joined #openstack-infra17:05
*** gokrokve has quit IRC17:05
*** marcoemorais1 has quit IRC17:06
*** amcrn has quit IRC17:06
*** marcoemorais2 has joined #openstack-infra17:06
*** marcoemorais2 has quit IRC17:06
*** amcrn_ is now known as amcrn17:06
*** tsg has quit IRC17:07
*** marcoemorais1 has joined #openstack-infra17:07
nibalizermordred: wb!17:07
*** gondoi is now known as zz_gondoi17:07
*** paulrad has joined #openstack-infra17:08
*** bdpayne has joined #openstack-infra17:08
*** dkranz has joined #openstack-infra17:09
*** bdpayne has quit IRC17:10
*** bdpayne has joined #openstack-infra17:10
*** amotoki has quit IRC17:10
*** palar has joined #openstack-infra17:11
*** yjiang5 has joined #openstack-infra17:12
*** paulrad has quit IRC17:12
*** marun_ has joined #openstack-infra17:14
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for critical keystone regression
*** Ryan_Lane has joined #openstack-infra17:14
*** flaviof is now known as flaviof_zzz17:14
*** chuckC has quit IRC17:16
*** mwagner_lap has joined #openstack-infra17:16
jeblairrussellb: for 113349, do you have plans to add the f20 preview node type to hpcloud?17:16
openstackgerritA change was merged to openstack-infra/config: Start running bashate on config (nonvoting)
jeblairsdague, dtroyer: ^ that changes the way bashate runs on devstack17:18
*** yjiang5 has quit IRC17:18
*** habib has quit IRC17:19
pelix1so I identified the root cause of what in broke the compare-xml, but since that was reverted, I have to use a new changeId. Should I include the old changeId into the commit message in the body so that there is a reference back to the original?17:19
*** pelix1 is now known as pelix17:19
openstackgerritA change was merged to openstack-infra/config: f20 should run full tempest runs
clarkbpelix: sure, it may help reviewers and historians down the line17:20
*** Ajaeger has joined #openstack-infra17:20
*** harlowja_away is now known as harlowja17:20
pelixclarkb: sure just wondered if there was a standard convention17:20
AjaegerI see with glance_store a problem uploading to pypi:
Ajaegerfungi, clarkb: Any ideas about this error "403 Client Error: You are not allowed to edit 'glance-store' package information"17:21
clarkbpelix: I don't think so, generally if you are undoing a revert with edits I would refer back to the origianl change(s)17:21
clarkbAjaeger: pypi user permissions are probably not correct I will check shortly17:21
Ajaegerclarkb: thanks!17:21
clarkbjeblair: nibalizer: we should try to get changes like in soonish, as that will allow us to keep moving on the puppet3 front right?17:22
nibalizerclarkb: thats not a blocker17:22
nibalizerits a good thing to ahve17:22
nibalizerbut its just noise in the logs, not hard failure17:22
clarkbnibalizer: oh ok17:22
clarkbnibalizer: the stack for the puppet3 master has a bunch of those type of changes at the bottom though17:23
*** yjiang5 has joined #openstack-infra17:23
nibalizerim totally with you on not touching gerrit with a ten foot pole unless we have to17:23
clarkboh but this is gerrit specifically ok17:23
nibalizeryea, i did that because there was a lot of red and I couldn't easily differentiate between warn and error17:23
nibalizerand we should merge that, if we can17:23
clarkbAjaeger: so the sdist worked
clarkbAjaeger: this could be a twine bug perhaps?17:24
Ajaegerclarkb: I don't follow - no clue about twine...17:24
*** gema has joined #openstack-infra17:25
clarkbAjaeger: twine is the utility we use to upload to pypi. dstufft wrote it. Basically the utility is that you don't have to run arbitrary code via python and get more consistency this way17:25
Alex_Gaynorclarkb, Ajaeger: I seem to recall twine has a bug with packages that have -s in their names /cc dstanek17:25
Alex_Gaynorerr, dstufft17:25
clarkbAlex_Gaynor: huh that sounds like a funny bug17:25
Ajaegerclarkb, Alex_Gaynor Thanks for the explanation.17:25
Alex_GaynorActually, looks like it was underscores:
openstackgerritAdam Gandelman proposed a change to openstack-infra/devstack-gate: Set required devstack variables for Ironic+grenade
AjaegerAlex_Gaynor: "_s"17:26
*** Sukhdev has quit IRC17:26
Alex_GaynorCan someone tell dstufft it's his job to fix this bug, it's been annoying us for cryptography as well :-)17:27
Ajaegerdstufft: ping17:27
openstackgerritA change was merged to openstack-infra/config: Add f20-devstack-virt-preview node type
*** yjiang5 has quit IRC17:28
*** yolanda has quit IRC17:28
clarkbAjaeger: thankfully the sdist should work just fine for now17:28
jeblairwell, it has impacted openstack now :/17:28
clarkbAjaeger: so I think we can get away with the current state, but should fix that twine bug17:29
Ajaegerclarkb: good to know17:29
Ajaegerclarkb: should I file a bug or is the one on twine enough?17:29
clarkbI am inclined to say the one on twine is enough, unless we want to make changes to workaround that bug on our side17:30
russellbjeblair: re f20 node type, i certainly can, was just keeping it small since it's an experimental thing to start with17:30
*** cipcosma has quit IRC17:30
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Adds Ironic parallel Tempest job
nibalizerhey Alex_Gaynor, anita asked me to take over your livegrep work since it seems to have stalled out, is that okay with you?17:30
russellbjeblair: i'm also working hard in the background to see if i can get some things going in the centos community to allow this to eventually be centos 7 based instead17:30
clarkbadam_g: that reminds me, how is the sideways upgrade going?17:30
russellbfyi i guess17:30
Alex_Gaynornibalizer: thats fine; sorry I haven't had more time for it, let me know if there's anything I can do17:30
Ajaegerclarkb: ok17:30
nibalizerAlex_Gaynor: sure, i just didn't want to step on toes17:31
krotscheckCan someone give me a pointer why gate-config-layout died on this review? It seems like a really innocuous change….
jeblairrussellb: no problem.  we'll just want to make sure it exists on hpcloud before we make it not-experimental.  also ++ on centos7, that means we can continue to test those features consistently throughout the openstack release lifecycle17:31
Alex_Gaynornibalizer: not at all; thanks for running with it!17:31
*** imeerovi has joined #openstack-infra17:31
*** SumitNaiksatam has joined #openstack-infra17:31
nibalizerkrotscheck: gate-config-layout was just derping at that time17:31
jeblairrussellb: also, i -1d the change to add the job because it went into silent, not experimental.  otherwise all looks good17:31
nibalizergive it a recheck and you should be ok17:31
clarkbkrotscheck: there was a bad commit to JJB that broke that job. It has been reverted so you should be fine to recheck17:31
jeblairclarkb: where did we end up on the firewalld question?  11388417:32
jeblairclarkb: now that devstack has it fixed, do we also want to yank it from our base images, or do we care?17:32
krotscheckclarkb: Derp… Did recheck stop working?17:33
*** yjiang5 has joined #openstack-infra17:33
clarkbkrotscheck: no it may just take some time with the feature freeze backlogs17:33
clarkbjeblair: I decided it was a harmless change that would give us consistency in our images17:33
clarkbjeblair: but didn't realize that devstack had corrected on that side when I +2'd17:33
clarkbjeblair: so I think we can probably just abandon that change17:34
*** mfink has joined #openstack-infra17:34
jeblairkrotscheck: we also don't leave 'starting check jobs' comments anymore17:34
nibalizerkrotscheck: yea what jeblair said17:35
*** e0ne has quit IRC17:35
nibalizerkrotscheck: also clarkb and i are looking sideways at this node.js app called 'subway' that does web irc client w/ persistent session17:35
*** skath_ has left #openstack-infra17:35
nibalizerhave you heard of it?17:35
*** ashaeron has joined #openstack-infra17:36
adam_gclarkb, its going okay. im hacking in things required to be set by DSG as a patch ontop of the main grenade patch (which still needs review)
adam_gi mean
*** paulrad has joined #openstack-infra17:37
*** terryw has quit IRC17:37
adam_gthere seems to be an issue with grenade and ENABLE_HTTPD_MOD_WSGI_SERVICES=True17:37
krotschecknibalizer: You lost me at “node.js app"17:38
*** annegent_ has joined #openstack-infra17:38
adam_gwhich AFAICS was not something that stable/icehouse ever deployed, so this is the first time grenade is seeing it.17:38
*** alexpilotti has quit IRC17:39
clarkbadam_g: now that I am looking at that change again (116761) when you modify the localrc to prevent exercising the old side, should you be modifying $BASE/old/grenade/localrc instead of $BASE/new/grenade/localrc?17:39
nibalizerkrotscheck: lulz :D17:40
clarkbadam_g: I am not sure which side exercising stuff falls on17:40
clarkbsdague: ^17:41
krotschecknibalizer: And then I looked it up and the readme lost me at “mongo”17:41
adam_gclarkb, thats not clear to me either :|  i dont see any reference to $BASE/old/grenade/localrc17:41
*** alexpilotti has joined #openstack-infra17:41
*** alkari has quit IRC17:41
clarkbkrotscheck: mongo isn't used anymore17:41
*** alkari has joined #openstack-infra17:41
*** paulrad has quit IRC17:41
clarkbkrotscheck: and I don't see it in the readme17:42
adam_gclarkb, but things that would be required by the 'old' side get written to $BASE/new/grenade/localrc (ie, BASE_*), which made me think $BASE/new/grenade/localrc is the single localrc used by grenade (there isn't an old/grenade and new/grenade)17:42
clarkboh I see17:42
clarkbbecause grenade has a complete view17:42
clarkband old vs new is just for the things being upgraded17:42
adam_gyeah. actually17:43
*** annegent_ has quit IRC17:43
adam_gthe slave runs grenade from $BASE/new/grande17:43
clarkbI am +2 on the d-g change17:44
clarkbI don't have any ideas on the wsgi stuff17:44
clarkbMOD_WSGI_SERVICES is keystone and horizon?17:44
adam_gclarkb, im probably going to keep that change updated iwth other things i may find that need to get set.17:44
adam_gyeah, but keystone is what is failing to start on the new side17:44
clarkbmorganfainberg: ^ maybe you have some ideas17:44
clarkbadam_g: that works for me17:45
clarkbadam_g: maybe WIP it until you are happy with it?17:45
adam_gi saw this locally and punted on it and disabled it, since i saw no other grenade run is using MOD_WSGI yet17:45
*** hdd has joined #openstack-infra17:46
*** dane_leblanc has joined #openstack-infra17:46
clarkbadam_g: with wsgi disabled are you able to do an upgrade and now it is a matter of getting the tests to pass?17:46
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Make JJB python 3 compatible
clarkbpelix: why did the popitem fail?17:47
clarkbpelix: do we know?17:47
*** pcrews has quit IRC17:47
pelixBecause we shouldn't have used it :)17:47
*** ashaeron has quit IRC17:47
adam_gclarkb, with wsgi enabled, devstack fails to start keystone on the new side (or at least, its curl check of the service fails, the apache log actually looks okay). it works locally when disabled, checking in experimental now17:47
pelixit removed data from the original dict that gets reused on subsequent iterations since the dicts are never copied in between17:48
clarkbpelix: oh so it was "recursively" digging to the bottom of a dict17:48
clarkbpelix: then it failed17:48
clarkbwhereas the iterator will raise iteration exception and we continue properly17:48
adam_gclarkb, beyond that, there are two tempest patches are +A'd and be merging today, that should get tests passing entirely17:49
clarkbadam_g: nice!17:49
pelixNot quite, it was operating on the source dict defining the job group. If the dict was copied, both approaches work, but since it's using references upon a second usage, it wouldn't find the required job-group17:49
adam_gclarkb, the biggest blocker now it seems is getting the grenade patch reviewed @
fungijeblair: we're getting a ton of "Exception: Manager rax-foo is no longer running" from nodepoold (for all three rax regions)17:50
jeblairfungi: probably related to the nodepool config change i just approved17:50
fungi"ERROR nodepool.ProviderManager: Unable to list servers while waiting for xxx will retry"17:50
clarkbpelix: gotcha, thanks17:51
krotscheckclarkb: So, using “recheck” in a comment is supposed to clear the jenkins -1, no?17:51
clarkbkrotscheck: not anymore17:51
clarkbkrotscheck: we removed the noise of reseting the vote17:51
clarkbkrotscheck: that was an implementation detail for the freshness checks which we aren't doing17:51
jeblairkrotscheck: care to elaborate?17:52
krotscheckWell, usually, when a user does something, they want some indication that the thing they did was useful. But now adding “recheck” to a review provides no user indicator that Things-Are-Happening(TM)17:52
*** morganfainberg is now known as morganfainberg_Z17:53
*** arnaud has quit IRC17:53
jeblairkrotscheck: leaving comments is a terrible user interface for something like this.  we should neither encourage nor leave more comments ourselves than are absolutely necessary17:53
fungijeblair: oh, yep, started complaining at 17:40:37 so that was probably right about when it got applied17:53
*** tonytan4ever has joined #openstack-infra17:54
fungitimestamp on nodepool.yaml confirms17:54
krotscheckjeblair: Ok, so if I want to recheck, and I shouldn’t leave a comment, how do I do that/17:54
jeblairkrotscheck: it's not possible yet, because no one has written an alternate interface.  in the mean time, we try to minimize it.  because comments are there fore people to talk about the change17:54
*** koolhead_ has quit IRC17:55
jeblairkrotscheck: if you would like to enhance zuul to support out-of-band enqueing, it would be a welcome change17:55
krotscheckjeblair: Ok, then how does the UI indicate to me that my recheck request has been received and is being processed?17:55
jeblairkrotscheck: we ran just fine without spamming 'starting' comments for a few years.  if you're really curious, you can check and if you would prefer more immediate feedback, then assistance with jheskeths changes to get the zuul status page in order so that it can be used on the review page would be helpful17:56
jeblairkrotscheck: i am aware that feedback to users is good.  but the costs in this case are high and other concerns outweigh them.17:58
krotscheckSure! I’ll put that right after getting infra on storyboard :)17:58
jeblairkrotscheck: that's the spirit :)17:58
* krotscheck wouldn’t mind working on something other than storyboard for a while.17:58
jeblaircross pollination++18:00
clarkbfungi: need any help with debugging that?18:00
jeblairfungi: i think the behavior may be that nodes that were being processed will throw those ecxeptions until their processing times out (about 1h)18:01
fungijeblair: oh! the manager with which they were associated has been replaced18:02
*** aj_ has joined #openstack-infra18:02
fungiyeah, i think that explains it18:02
*** tsg has joined #openstack-infra18:03
*** weshay has joined #openstack-infra18:03
*** tgohad has quit IRC18:04
jeblairfungi, clarkb: we should probably raise a more specific exception there, and use it to escape some of the loops that we're currently spinning in18:05
jeblairsince those loops are designed to mask transient errors, and this is not one18:05
clarkbsounds good18:05
*** radez is now known as radez_g0n318:06
fungimakes sense18:06
*** mriedem1 is now known as mriedem18:07
*** marcoemorais1 has quit IRC18:08
*** aj_ has quit IRC18:08
*** Mithrandir_ is now known as Mithrandir18:08
*** marcoemorais has joined #openstack-infra18:09
*** marun_ has quit IRC18:09
*** e0ne has joined #openstack-infra18:10
bswartzI have a question18:10
bswartzhow can I create tags for a stackforge project?18:10
bswartzI want to create the juno-3 tag for manila18:11
*** shayneburgess has joined #openstack-infra18:11
clarkbbswartz: there is a gerrit ACL that needs to be set up if it isn't already, then members of the group given these permissiosn can push tags to gerrit with `git push gerrit tagnamehere`18:11
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Don't retry forever when a provider is stopped
bswartzclarkb: where can I see the existing acls?18:12
clarkbbswartz:,access there18:12
jeblairfungi, clarkb: ^ i think that should do it.  more generally, i think nodepool is violating some early assumptions about how providers would work and probably needs a bit of an overhaul there.18:12
*** palar_ has joined #openstack-infra18:12
clarkbbswartz: the manila release group has the ability to push tags18:12
clarkbbswartz: things to keep in mind. Make sure that gerrit knows about the commit you have tagged and that it has already merged to a branch18:12
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Define -T/--no-topic to disable review submit with topic
clarkbbswartz: sign your tags, and treat them as immutable (you don't want to move a tag later)18:13
fungibswartz: see also
bswartzclarkb: when I tried to tag I got a gpg error18:13
bswartzIs GPG a requirement?18:13
fungibswartz: very much so, yes18:13
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: -F/--force-rebase has no effect if rebase is disabled by config
bswartznever needed gpg for anything else18:13
bswartzI'll figure it out18:14
fungibswartz: also we want to rename your git repositories from stackforge to openstack sometime soon. vponomaryov suggested infra should just pick a time convenient for us and let you know when that's happening, perhaps via a thread on the -dev ml tagged [manila]18:15
bswartzyes I'm aware of that18:15
*** palar has quit IRC18:15
fungiokay, awesome18:15
bswartzI figure it's better to do the rename after J-318:15
fungiyep, we have to take zuul and gerrit offline briefly for the rename, so we too will want to wait until after the milestone rush has subsided18:16
bswartzregarding gpg, is there a place I have to register my public key?18:16
fungibswartz: i wrote this which may help you...
*** mspreitz has joined #openstack-infra18:17
*** annegent_ has joined #openstack-infra18:18
*** jlk_clone has joined #openstack-infra18:18
*** masayukig_ has quit IRC18:18
fungii suppose now that we have that, i can link it in the tagging instreuctions instead of the fedoraproject link currently there18:18
*** jlk_clone is now known as jlk18:19
*** jlk has joined #openstack-infra18:19
jeblairjesusaurus: on 115116 my math indicates a fator of 16 reduction in sample storage, not 30.18:20
*** radez_g0n3 is now known as radez18:21
*** HeOS has quit IRC18:21
*** ZZelle_ has joined #openstack-infra18:22
*** adalbas has quit IRC18:22
*** habib has joined #openstack-infra18:22
*** ashaeron has joined #openstack-infra18:23
*** masayukig has joined #openstack-infra18:25
jesusaurusdid i derp that math? i might've18:25
ekarlsophschwartz: around ?18:26
jeblairjesusaurus: i come up with 19206 samples for your proposal, compared to 275220 under current config18:27
fungiekarlso: he's apparently on a family vacation all this week18:27
ekarlsodarn it18:27
*** arnaud has joined #openstack-infra18:27
*** marcoemorais has quit IRC18:28
jeblairjesusaurus: 360+10080+8766=1920618:28
jeblairjesusaurus: let me know if you think i did something wrong; i have a spreadsheet up calculating alternatives :)18:28
*** pcrews has joined #openstack-infra18:28
*** ashaeron has quit IRC18:28
jesusaurusjeblair: ah, i was just doing some back-of-the-napkin comparisons of 10-minute for 5 years vs 1-hour for 1-year and assumed the others were negligible18:28
*** marcoemorais has joined #openstack-infra18:28
jesusaurusthat was probably a bad assumption on my part18:29
*** gokrokve has joined #openstack-infra18:29
*** marcoemorais has quit IRC18:29
jeblairjesusaurus: they are actually -- only changing the year is 0.07x vs 0.06x18:29
jeblairthat was probably incomprehensible let me try again18:29
*** marcoemorais has joined #openstack-infra18:29
jeblairjesusaurus: they are actually -- only changing the 5y to 1y retention is 0.07x compared to 0.06x if you change that and also the 6h-1h retention18:30
jesusaurusi dont follow where 0.07x and 0.06x are coming from18:31
*** gokrokve_ has quit IRC18:31
jeblairjesusaurus: sorry, i inverted the math because i prefer to think the other way :)  19206/275214=0.069...18:33
*** palar_ has quit IRC18:33
jeblair6.9% of current usage18:33
*** gokrokve has quit IRC18:33
*** rushiagr is now known as rushiagr_away18:33
jesusaurusthe way i got 30 is: 5yr to 1yr is a factor of 5; 10minute to 1hour is a factor of 6. 5*6 gives me 30. very rough math on my part, and i wouldnt be surprised if i didnt factor in something significant18:34
*** adalbas has joined #openstack-infra18:34
*** emagana has quit IRC18:34
*** emagana has joined #openstack-infra18:35
*** daxatlas has quit IRC18:35
*** habib has quit IRC18:35
*** emagana has quit IRC18:36
*** emagana has joined #openstack-infra18:36
jeblairjesusaurus: ah, yes.  that's correct (it gives you the right number for the yearly metric), but at that point, that metric then accounts for ~50% of the total storage, so that explains why you're off by about 2x.18:37
clarkbfyi logstash doesn't seem to be indexing all of the data I expect it to be indexing (details over in -qa) so I am going to poke at that after lunch18:39
jesusaurusso yeah, assuming the others were negligible was an error on my part18:39
jeblairokay, i'll have some alternate proposals on that review in a minute18:40
*** flaviof_zzz is now known as flaviof18:40
*** _nadya_ has joined #openstack-infra18:41
jesusaurusclarkb: anything i can help with?18:41
clarkbjesusaurus: possibly. I need to dig through logs first to see why it is failing but basically screen-n-cond.txt isn't ending up in logstash but the old screen n cond for grenade does18:41
clarkbjesusaurus: so I don't think it is a parser failure, probably something earlier in the pipeline18:42
clarkbjesusaurus: I will let you if/when I find anything18:42
*** zul has quit IRC18:43
*** lttrl has quit IRC18:43
*** annegent_ has quit IRC18:44
*** _nadya_ has quit IRC18:46
*** e0ne has quit IRC18:47
*** zul has joined #openstack-infra18:48
*** amcrn_ has joined #openstack-infra18:51
*** amcrn has quit IRC18:54
*** bswartz has quit IRC18:54
*** bswartz has joined #openstack-infra18:55
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: ../../message
*** bswartz has quit IRC18:56
*** bswartz has joined #openstack-infra18:56
jeblairjesusaurus: okay, left comments is 11511618:56
jesusaurusjeblair: thanks18:56
jeblairnotmyname: interested in in 115250 ?18:58
*** amuller__ has quit IRC18:58
notmynamejeblair: oh. umm...just what I said in the comments. is there a discussions somewhere else?18:59
jeblairnotmyname: oh did i miss your comments?18:59
openstackgerritJeremy Stanley proposed a change to openstack-infra/git-review: Work toward Python 3.4 support and testing
notmynamejeblair: I see it at the bottom (aug 27)19:00
sdagueadam_g: is there a reason is WIP?19:01
jeblairnotmyname: whoops, sorry about that19:01
notmynamejeblair: no worries19:01
notmynamejeblair: can you give any insight or context for
adam_gsdague, still fleshing out exactly whats needed in devstack-gate, based on attempts at getting the sideways job running in experimental (
*** derekh has joined #openstack-infra19:03
sdagueadam_g: ok19:03
sdagueadam_g: are there other patches you feel are ready for review related to the ironic sideways effort?19:03
adam_gsdague, the main grenade patch @
adam_gsdague, i just put up two fixes for to devstack and grenade that are relevant but can be worked around in the meantime by disabling mod wsgi19:04
uvirtbotLaunchpad bug 1365105 in grenade "grenade fails to upgrade keystone when mod wsgi is enabled" [Undecided,In progress]19:04
*** e0ne has joined #openstack-infra19:05
*** Ajaeger has quit IRC19:06
adam_gsdague, all required tempest changes are now +A'd. would help with related tempest work outside of grenade19:06
sdaguenova_bm_create_node_and_port is something I envisioned would actually be mysql < ironic-dump.sql19:08
jeblairnotmyname: i'd recommend mordred or clarkb; they are more thoroughly versed in the babel machinery.19:08
pelixin between the comments on various patches I'm seeing conflicting requirements. Previous comments indicate I should follow the hacking guide at but at the same time I'm being asked to move an existing dynamic import of a std lib module after the p19:08
*** dmellado has quit IRC19:08
sdagueadam_g: is there a reason for doing mysql inserts inline?19:08
notmynamejeblair: thanks. just curious about hwere it came from19:08
*** otherwiseguy has joined #openstack-infra19:08
*** derekh has quit IRC19:08
adam_gsdague, we're not actually deploying nova-bm and have no API server to accept the data otherwise. we cant really ship a database dump, as we need the nova-bm node inventory to reflect the 'baremetal' VMs we'll end up using post-migration19:09
pelixIs there a convention, is it alright in this case to move imports not part of your patch, and should I move a std lib import to the end of all the imports because it's a dynamic import as opposed to a static one, or should I keep it grouped with the other std lib imports?19:09
*** paulrad has joined #openstack-infra19:09
* clarkb sees he just got volunteered for a thing19:10
mordrednotmyname: we copied most of the babel stuff from swift, fwiw :)19:11
*** dmellado has joined #openstack-infra19:11
clarkbnotmyname: re 97092, I believe we need/want those files to be inlcuded in our sdists. We get that cheap in that location iirc19:11
clarkbnotmyname: the exact location probably doesn't matter, but if we are consistent it makes the packaging and installation of .po(t) files much easier19:11
adam_gsdague, also, even if devstack could set us up a nova-bm API server, it would have been stopped by the time that migration stuff ran19:11
mordredalso, what clarkb says19:12
sdagueadam_g: ok, that's sufficient explanation for me19:12
*** radez is now known as radez_g0n319:15
*** e0ne has quit IRC19:16
clarkbpelix: I would keep it grouped, I blieve that is what the rest of openstack tries to do19:17
clarkbpelix: unless you need it to be dynamic at post "parse" time19:17
pelixnope, just at parse time19:17
pelixclarkb: thanks19:19
*** e0ne has joined #openstack-infra19:19
openstackgerritDarragh Bailey proposed a change to openstack-infra/git-review: Enable color support based on tty and config
*** homeless has quit IRC19:19
*** gokrokve has joined #openstack-infra19:19
*** homeless has joined #openstack-infra19:19
pelixclarkb: apparently in neutron dynamic imports are done afterwards, but I'm guessing that's not standardized?19:20
*** emagana has quit IRC19:20
clarkbpelix: huh I thought hacking checked that. maybe hacking doesn't deal with them19:20
*** zeus has quit IRC19:21
*** zeus has joined #openstack-infra19:22
*** zeus has quit IRC19:22
*** zeus has joined #openstack-infra19:22
*** marun_ has joined #openstack-infra19:24
openstackgerritMatt Riedemann proposed a change to openstack-infra/devstack-gate: Deprecate API_RATE_LIMIT setting for Tempest runs
*** sabeen has joined #openstack-infra19:28
*** _nadya_ has joined #openstack-infra19:29
pelixclarkb: no worries, I've just asked that someone make a clear statement in the review as to whether the suggestion from the reviews should override the documented style. just one of those cases that opinion doesn't work well19:31
clarkbpelix: ya I am fine with it either way really19:31
*** _nadya_ has quit IRC19:33
*** _nadya_ has joined #openstack-infra19:34
*** ociuhandu has quit IRC19:34
*** jergerber has joined #openstack-infra19:35
*** homeless has quit IRC19:38
*** Sincler has quit IRC19:39
*** IAX has quit IRC19:39
*** gargola has quit IRC19:39
*** _nadya_ has quit IRC19:39
*** hashar has joined #openstack-infra19:39
*** gargola has joined #openstack-infra19:39
*** homeless has joined #openstack-infra19:40
*** IAX has joined #openstack-infra19:40
*** Sincler has joined #openstack-infra19:40
*** paulrad has quit IRC19:41
*** che-arne has quit IRC19:41
*** paulrad has joined #openstack-infra19:41
*** marun_ has quit IRC19:42
openstackgerritAndrey Kurilin proposed a change to openstack/requirements: restrict requests version with !=2.4.0
*** paulrad has quit IRC19:45
*** weshay has quit IRC19:46
*** paulrad has joined #openstack-infra19:46
*** sballe has joined #openstack-infra19:47
clarkbbefore I grab lunch, when do elections happen? is that real soon now?19:48
*** hashar has quit IRC19:49
*** paulrad has quit IRC19:50
*** radez_g0n3 is now known as radez19:51
*** marun_ has joined #openstack-infra19:55
pleia2mm lunch, might need to find some place outside, it's september so condo is starting to get toasty in the afternoon19:55
anteayaSergeyLukjanov: there is no correct specific recheck syntax for third party: timestamp: 19:48:38 <jeblair> #topic  comment syntax for third-party CI recheck (dougwig, from third-party CI meeting)19:58
*** tgohad has joined #openstack-infra19:59
*** tsg has quit IRC19:59
*** chuckC has joined #openstack-infra19:59
anteayappai: what do you want me to look at with I am already +1, I can't get anymore +1 than I already am19:59
anteayaI'm back and doing poorly20:00
anteayalet me know if there is anything i need to address20:00
pleia2anteaya: rest :)20:00
jeblairfungi, clarkb: i'm going to do the mailing list rename now20:01
fungilet me know if you need help20:01
pleia2thanks jeblair20:01
openstackgerritA change was merged to openstack-infra/reviewstats: Add lxsli and jonpaul-sullivan to tripleo core
dhellmannThe oslo team is considering a FFE for the AMQP 1.0 driver, esp. since it is optional. However, we'd like to ensure there isn't going to be a major issue with getting real testing in place. kgiusti has put together but it's not clear if that approach is acceptable to the infra/qa teams. Can I get some feedback on the ML thread?20:02
jeblairi'm single-approving the changes because i think we all agreed on the process20:02
pleia2I'll follow up on the mailing list thread once we know all is working20:02
*** paulrad has joined #openstack-infra20:03
*** tsg has joined #openstack-infra20:03
*** annegent_ has joined #openstack-infra20:05
clarkbdhellmann: iirc ianw has a change up to make it possible. I don't expect any major issues20:05
dhellmannclarkb: ok, great, that's what I wanted to make sure of20:06
clarkb115752 should probably depend on the addition of the bare-f20 nodetype though20:06
*** radez is now known as radez_g0n320:06
clarkbianw: ^ do you expect problems? I know the postgres thing was something you ran into. crinkle says we should see a release of that module this week20:06
*** arosen has joined #openstack-infra20:06
*** e0ne has quit IRC20:06
jeblairdhellmann: main thing to know about non-centos/lits node types is that we can't test on them for the full openstack lifecycle, so when they go out of support, we will drop them and their associated jobs immediately20:07
clarkbianw: and if they don't release you tested the work around and I am willing to roll with that20:07
dhellmannwhen the OS goes out of support, you mean?20:07
jeblairdhellmann: yep20:07
*** tgohad has quit IRC20:07
*** stevemar has quit IRC20:07
dhellmannjeblair: ok, that's good to know. I would expect the use of f20 to be temporary until the packages needed are available for the centos/lts nodes to consume.20:08
jeblairdhellmann: sounds like a plan20:08
anteayaI just took a copy of the subscriber list of -request, just in case20:08
arosenhi, I'm kinda stuck on this tox error. It's telling me: --- import errors --- congressclient.tests.v1.test_policyNon-zero exit code (2) from test listing. If I debug it like i normally do i.e: source .tox/py27/bin/activate; i can import all the imports on this test so I'm not quite sure what the problem is.20:08
arosenAny ideas on how to debug this further? nosetests and python testr both pass.20:09
openstackgerritA change was merged to openstack-infra/config: Removes mailing list third-party-request
clarkbarosen: so `import congressclient.tests.v1.test_policy` succeeds in an python interpreter run out of that venv?20:10
*** marcoemorais has quit IRC20:10
*** marcoemorais has joined #openstack-infra20:11
*** marcoemorais has quit IRC20:12
arosenclarkb:  yup :/
openstackgerritCedric Brandily proposed a change to openstack-infra/git-review: Define -T/--no-topic to disable review submit with topic
*** marcoemorais1 has joined #openstack-infra20:13
clarkbarosen: interesting try `.tox/py27/bin/python -m discover -t ./ . --list`20:15
clarkbarosen: that is the command that is failing but testtools instead of subunit so that the output comes out human readable. And let us see if that is more verbose20:15
arosenclarkb: it just says failed to import :/
arosenlet me try playing around with my testr.conf and see if that helps anything.20:17
*** fifieldt_ has joined #openstack-infra20:17
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1363103
uvirtbotLaunchpad bug 1363103 in neutron "OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO routerl3agentbindings" [High,Confirmed]
*** paulrad has quit IRC20:19
*** paulrad has joined #openstack-infra20:19
jeblairanteaya, fungi: puppet change is in place.  i'm stopping exim and performing the steps listed here (in part 2):
*** HeOS has joined #openstack-infra20:20
*** fifieldt has quit IRC20:21
* anteaya clicks on the linked wikipage20:22
*** mfink has quit IRC20:23
*** julim has quit IRC20:24
*** paulrad has quit IRC20:24
*** ashaeron has joined #openstack-infra20:25
arosenclarkb:  hrm same thing. Is there anyway to get a trace where the import failed :/20:25
jeblaircomplete, and restarted exim20:26
jeblairapproving change #220:26
clarkbarosen: usign the interpreter is what I do which is why I am surprised it doesn't work20:27
clarkbarosen: is this code available for others to reproduce?20:27
pleia2jeblair: looks good \o/20:27
jeblairin 10 mins, we'll see if puppet is happy with the list20:27
*** sballe has quit IRC20:28
arosenclarkb:  one sec i think i might have found out why with strace20:28
openstackgerritJeremy Stanley proposed a change to openstack-dev/hacking: Work toward Python 3.4 support and testing
openstackgerritJeremy Stanley proposed a change to openstack-dev/pbr: Work toward Python 3.4 support and testing
openstackgerritJeremy Stanley proposed a change to openstack-infra/elastic-recheck: Work toward Python 3.4 support and testing
openstackgerritJeremy Stanley proposed a change to openstack-infra/reviewstats: Work toward Python 3.4 support and testing
openstackgerritJeremy Stanley proposed a change to openstack-infra/storyboard: Work toward Python 3.4 support and testing
openstackgerritA change was merged to openstack-infra/config: Creates third-party-requests mailing list
arosenclarkb:  sHere's the patch:
openstackgerritAndreas Jaeger proposed a change to openstack-dev/cookiecutter: warn against sorting requirements
*** cdent has quit IRC20:32
*** kurtmartin has joined #openstack-infra20:33
openstackgerritJeremy Stanley proposed a change to openstack-dev/cookiecutter: Work toward Python 3.4 support and testing
*** mwagner_lap has quit IRC20:34
anteayamy heart is still tap-dancing from the anesthetic but I can't eat anymore apples or oatmeal20:36
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add Python 3.4 and PyPy to Ubuntu Trusty slaves
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add non-voting gate-git-review-python34 job
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Update StackForge documentation to recommend py34
*** cnesa10 has joined #openstack-infra20:37
pleia2 \o/20:37
crinkleclarkb: we're trying to release puppetlabs-postgresql but it will be a major version bump, so I don't know if that helps you20:37
dolphmthis fixes a transient and a critical regression in keystone - can someone prioritize it in the gate for us?
*** shayneburgess has quit IRC20:38
jeblairdolphm: can do20:39
dolphmjeblair: thank you20:40
*** shayneburgess has joined #openstack-infra20:40
anteayapleia2: your welcome email just got sent out from third-party-requests20:41
*** shayneburgess has quit IRC20:41
pleia2anteaya: I just resent it to the proper address20:41
jeblairdolphm: done20:41
pleia2so yay, works20:42
anteayapleia2: ah, well it worked20:42
clarkbcrinkle: I think ianw said that the module worked as is if we checked it out directly so major version bump may be ok20:42
dolphmjeblair: /salute20:42
clarkbcrinkle: we may need to tweak a few things, postgres is less scary to us because its only used on test slaves20:42
anteayapleia2: I'll try repling20:42
pleia2anteaya: followed up on dev and infra about the change too (same thread as initial announcement)20:42
jeblairhrm.  alien-list doesn't work on hpcloud because of the pseudo regions20:43
anteayapleia2: yay20:43
*** emagana has joined #openstack-infra20:43
*** tonytan4ever has quit IRC20:43
jeblairoh, but i can grep20:43
*** adalbas has quit IRC20:43
anteayapleia2: the reply works, yay and thanks for following up on -dev and -infra20:43
*** ildikov has quit IRC20:43
* pleia2 updates email filter20:44
pleia2all looks good20:44
fungijeblair: i use a delay between two and then filter for those which didn't un-alien in that span:
fungijeblair: and then i iterate over the resulting *.aliens lists for a nova delete20:47
*** sballe has joined #openstack-infra20:47
jeblairfungi: ah cool.  oh yeah, you said you were going to do that.  i won't step on your toes, unless you'd rather me do it.20:47
fungii have it running now, since you just reminded me20:48
fungii got sidetracked submitting 5 pages of py34 transition changes to gerrit20:48
jeblairfungi: wow, you did the stackforge projects?20:49
*** tonytan4ever has joined #openstack-infra20:49
fungii iterated over all of what was in gerrit. was easier that way20:49
fungiupdated the cookiecutters too20:49
fungii'be got a config change proposed to start installing python3.4 and pypy on trusty from the normal ubuntu packages, and a child change to add a non-voting job for it to git-review20:50
*** dprince has quit IRC20:50
*** shayneburgess has joined #openstack-infra20:50
*** yfried has quit IRC20:50
*** marcoemorais1 has quit IRC20:50
fungiand assuming that works, we can discuss whether we want to add non-voting jobs for everyone, or just switch them all from 33 to 34 and let them scream/fix whatever it breaks ;)20:51
*** doug-fish has joined #openstack-infra20:51
anteayafungi: and you have a -1 already20:52
fungiat least half the python33 jobs are non-voting now anyway, so we can probably safely just replace those. if the voting 33 jobs aren't too numerous i can pre-test them manually20:52
fungiaha, taskflow uses a tox.ini template/generator20:53
*** markmcclain1 has joined #openstack-infra20:53
anteayado you need py34 for havana?20:54
anteayahow much longer does havana have?20:54
*** markmcclain has quit IRC20:54
*** marcoemorais1 has joined #openstack-infra20:54
clarkbfungi: oh thank you for the py34 stuff20:54
*** ociuhandu has joined #openstack-infra20:54
fungianteaya: probably not, but i generated all the backport changes automagically20:54
anteayaah okay20:54
fungianteaya: if we ditch havana sooner than we ditch the precise-py3k nodes, then those changes can just get abandoned20:55
fungiand ultimately they're not (most of them anyway) needed for running python 3.4 jobs since the majority of those changes just edit the default list of environments. the ones which _do_ matter were the projects with "extra" stuff going on in their tox.ini to support 3.320:56
fungibut since i needed to identify those, it was easier to just do them all in a batch run20:56
* anteaya keeps an eye out for tox.ini files with extra stuff in them20:56
anteayabatches can be good20:57
clarkbfungi: were projects doing much more than PYTHONHASHSEED=0 or the occasional dependency difference?20:57
arosenclarkb:  ah found it!  I do:  from openstackclient.common import utils as osc_utils20:57
fungiclarkb: it was mostly py3k-specific dependencies, yeah20:58
arosenthat import isn't in the python-congressclient repo20:58
aroseni guess tox creates a source tarball and runs it's tests from that so that's what was missing.20:58
arosenthanks for your help though!20:58
clarkbyes by default that is how tox tests20:59
clarkbhowever your non sdist installed code would have the same issue in the virtualenv right?20:59
*** homeless has quit IRC21:00
*** emagana has quit IRC21:00
anteayafungi: this would be an example of a tox.ini file with extra stuff in it: yes?21:00
*** emagana has joined #openstack-infra21:00
*** homeless has joined #openstack-infra21:01
*** Sukhdev has joined #openstack-infra21:01
*** markmcclain1 has quit IRC21:01
openstackgerritA change was merged to openstack-infra/config: Edit the name of the third-party-requests ml
*** markmcclain has joined #openstack-infra21:01
fungianteaya: yeah, keystone does some interesting things there because of nose21:02
fungianteaya: oh, though that's actually a bad cherry-pick21:02
fungithe master version of that makes a little more sense21:02
* fungi fixes21:02
bdpaynejeblair Could you help me understand what I need to do with chanserv for the openstack-security op privs?21:03
anteayafungi: this would be the master version: yes?21:04
*** annegent_ has quit IRC21:04
fungianteaya: yes21:05
anteayafungi: the master and the icehouse version look the same to my eyes, what makes this a bad cherry-pick?21:05
anteayathe icehouse version shouldn't have the nose stuff?21:06
anteayaI don't understand21:06
fungianteaya: it duplicated logic from the py33 venv setup for the master tox.ini into the stable/icehouse (and havana too) tox.ini21:06
*** tsg has quit IRC21:06
fungianteaya: note that the py33 setup i was copying in the master tox.ini is not present in the stable ones21:07
*** elixor has joined #openstack-infra21:07
jeblairbdpayne: send '/msg chanserv help'21:07
jeblairbdpayne: /msg chanserv op #openstack-security bdpayne21:07
jeblairbdpayne: should do it21:07
* anteaya looks again at the py33 setup in the master and stable tox.ini files21:08
jeblairbdpayne: specifically related to banning folks, you may be interested in 'access +b'.  see 'help access' and 'help flags'21:09
openstackgerritEdward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally
bdpaynehey thanks, that worked21:09
*** aysyd has quit IRC21:09
jeblairbdpayne: np21:09
*** emagana_ has joined #openstack-infra21:10
*** zz_johnthetubagu has quit IRC21:10
*** emagana has quit IRC21:10
*** mfer has quit IRC21:11
*** zz_johnthetubagu has joined #openstack-infra21:11
*** zz_johnthetubagu is now known as johnthetubaguy21:11
*** miqui has quit IRC21:11
fungianteaya: the addition of the [testenv:py33] section wasn't added until after stable/icehouse so doesn't need a similar [testenv:py34] section in the backport either21:12
openstackgerritA change was merged to openstack-infra/config: Change wording and typos in third_party.rst
*** david-lyle has quit IRC21:12
openstackgerritA change was merged to openstack-infra/config: Make py33 job experimental for cloudkitty-dashboard
*** emagana has joined #openstack-infra21:13
*** elixor has quit IRC21:13
*** imcsk8 has quit IRC21:13
*** imcsk8 has joined #openstack-infra21:14
anteayafungi: I see the diff21:15
*** emagana_ has quit IRC21:16
anteayafungi: I'm having difficulty linking backward from that page to see the datestamp for that addition21:16
*** emagana_ has joined #openstack-infra21:16
openstackgerritA change was merged to openstack-infra/config: Fix deprecation warnings in exim module
openstackgerritA change was merged to openstack-infra/config: Conditionally manage puppetdb connection
*** melwitt has joined #openstack-infra21:17
*** tsg has joined #openstack-infra21:19
*** emagana has quit IRC21:19
greghaynesjeblair: Hey, got -1'd for reasons I dont understand and I was told that it might be a UI bug21:20
openstackgerritA change was merged to openstack-infra/config: Fix depercation warnings in templates for unatttended upgrades
clarkbgreghaynes: jeblair: the issue is you the cahnge failed the requirements check21:20
clarkbso -1. but the CI table doesn't show that failure because it reports back with a emssage21:21
greghaynesoh! I totally glanced over that becuase it wasnt green or red21:21
*** melwitt has quit IRC21:22
*** melwitt has joined #openstack-infra21:23
clarkbthe js regex may need to be relaxed to capture non binary results21:25
jeblairclarkb, fungi:
fungijeblair: uhh... weird21:27
clarkbI was about to say mirror problems again but no thats local fil://21:28
*** dhellmann is now known as dhellmann_21:28
clarkbperhaps the image was built with a bad repo in the cache?21:28
fungilocal disk i/o problem? are we seeing this widespread?21:28
openstackgerritAnita Kuno proposed a change to openstack-infra/config: I missed an 's'
fungidigging in logstash now21:28
anteayajeblair: I realized I missed an 's' in the docs ^^21:29
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add a missing 's' to third-party-requests
fungiclarkb: jeblair: bingo. all in rax-iad21:30
jeblairanteaya: i made a better commit subject for you :)21:30
anteayayou did, thank you21:30
fungiclarkb: jeblair: better still, it's been happening for a couple days, so we didn't get an image there today21:31
clarkbfungi: the image build failed in a similar way?21:31
clarkbfungi: sounds like fun :)21:31
*** krtaylor has quit IRC21:32
*** david-lyle has joined #openstack-infra21:33
*** doug-fish has quit IRC21:33
fungiclarkb: no, i mean it's been happening since before today's image would have built, so presumably we didn't get a new image today (which would likely have caused it to cease happening)21:33
clarkboh I see. I thought maybe you were talking about vm disk failures in that region, but you mean that particular things failed21:34
fungii suspect the image builds didn't happen either because of the alien infestation (which are in the process of being deleted now) or because they were interrupted by our nodepool restart around the same time21:34
fungiclarkb: just looking at message:"fatal: serious inflate inconsistency" AND tags:console for the past 7 days21:35
fungiclarkb: though fun tidbit, it only hit check-projects-yaml-upstream once21:35
*** annegent_ has joined #openstack-infra21:35
fungithe others are for gate-nova-python27, gate-neutron-python27, gate-ceilometer-pep8 add gate-tempest-docs21:36
fungiso maybe it's not a bad image, but something else going sour in iad21:36
fungifor example
*** annegent_ has quit IRC21:37
clarkbright because those other jobs will be using different repos21:37
*** annegent_ has joined #openstack-infra21:38
clarkbbad disk? bit flips in memory? the next intel fp bug?21:38
fungibut whatever the underlying cause, it's only in rax-iad and started some time yesterday21:38
jrollfungi: I'm just reading up, but what time approximately? 7 or 8 pm pacific, perhaps?21:41
*** tonytan4ever has quit IRC21:42
*** mfink has joined #openstack-infra21:42
clarkbjroll: was 1840 UTC today which is ~1140 pacific if I did my maths correctly21:42
jrollfungi: disk io issues?21:42
jrollI'm curious when we started seeing it21:43
*** emagana_ has quit IRC21:44
jrollthere was a largish deploy to iad last night21:44
* jroll does math21:44
clarkboh when it started21:44
* clarkb runs the query21:44
jrollbegan at 0200 utc this morning?21:44
jrollif my math is right21:44
clarkboh fungi has it21:44
jroll7pm pacific last night21:44
fungijroll: 09:36 utc was the earliest sighting21:44
jrolloh, huh, ok21:45
fungithat would be 02:00 pdt21:45
jrollstill bad, but at least not something we broke with a deploy :P21:45
openstackgerritA change was merged to openstack-infra/config: Run requirements jobs for openstack-doc-tools
openstackgerritA change was merged to openstack-infra/config: Consolidate tripleo/os-apply-config ACL files
*** mriedem has quit IRC21:46
fungilocal time in iad would have been 05:00 (edt)21:47
openstackgerritA change was merged to openstack-infra/config: Remove /etc/puppet/modules on apply test
jrollI can't imagine a code change would cause corrupted memory, though, either21:47
clarkbjroll: did you perhaps deploy onto new machines?21:48
clarkband maybe one of them is having hardware trouble?21:48
jrollclarkb: 99% sure that's a negative21:48
jrollbut, old machines could be having issues21:48
jrollclarkb: any way we have instance ids for these VMs?21:49
clarkbits such an odd error, and it doesnt' affect a specific repo which makes me think it must be more of a general issue21:49
clarkbjroll: we can dig them up21:49
*** kgiusti has quit IRC21:49
jrollclarkb: I'd be interested to see if they were all on the same host or something21:49
*** emagana has joined #openstack-infra21:49
openstackgerritA change was merged to openstack-infra/config: Disable notifications in storyboard
jrollclarkb: if you think it's worth the effor, anyway21:50
jrollI'm not 100% sure I can dig up old instances, but I can try21:50
*** flaviof is now known as flaviof_zzz21:50
*** andreaf has quit IRC21:51
fungialien nodes total before deletion was 52, though 18 of those were in iad21:51
fungiwhich explains the over-quota errors nodepool was getting21:51
openstackgerritA change was merged to openstack-infra/config: Fix typo - run gate/check against oslo.serialization-py33
openstackgerritA change was merged to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension.
openstackgerritA change was merged to openstack-infra/config: Fix Berkshelf 3.x GeCode 3.x dependency
clarkbjroll: ya I can find a few now21:53
jrollclarkb: cool, let me know21:53
*** paulrad has joined #openstack-infra21:54
openstackgerritA change was merged to openstack-infra/config: Add #openstack-operators to the eavesdrop bot
clarkbjroll: d6b3da89-df68-4284-9ec5-bd0fa35a321e 622e322a-95c6-4f88-873c-e11382e621dd 80ae393d-35c4-43b3-9d82-2745c7c73090 0ed34e38-f8ce-421b-a1eb-ac7dc20862ee f9626b38-31d7-4735-9d7c-78db13904ca321:58
*** datsun180b has quit IRC21:58
* jroll looks21:58
*** paulrad has quit IRC21:59
*** baoli has quit IRC22:00
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add integration apply test
jrollclarkb: heh, nice one, all the same host22:03
jrollclarkb: I'll poke people internally, thanks for tracking that down :)22:03
clarkbjroll: thank you for checking on your side22:03
openstackgerritA change was merged to openstack-infra/config: Turn off manpage building
openstackgerritA change was merged to openstack-infra/config: Doc cleanup
*** Clabbe has quit IRC22:04
*** pballand has quit IRC22:04
*** marcoemorais1 has quit IRC22:05
*** marcoemorais has joined #openstack-infra22:06
*** shardy is now known as shardy_z22:07
*** openstack has joined #openstack-infra22:11
clarkbbut first trying to figure out if this is a legit derp or if we just need more fds22:11
*** eharney has quit IRC22:12
clarkbit actually looks like it may be leaking PIPEs22:13
clarkbwhich are used for the crm114 communication22:14
clarkbso bumping the limit is not so good22:14
*** emagana has quit IRC22:15
openstackgerritJeremy Stanley proposed a change to openstack-infra/nodepool: Don't retry forever when a provider is stopped
*** paulrad has joined #openstack-infra22:17
clarkbfungi: we both +2'd :)22:17
clarkbI can approve22:17
fungiclarkb: feel free22:17
*** sdake_ has joined #openstack-infra22:18
*** dkranz has quit IRC22:19
*** dkranz has joined #openstack-infra22:20
*** erlon has quit IRC22:20
*** markmcclain has quit IRC22:20
*** mfink has quit IRC22:20
openstackgerritA change was merged to openstack-infra/config: Add readthedocs for heat-translator
openstackgerritA change was merged to openstack-dev/pbr: use get_boolean_option for warnerrors
fungiwe're a little over 4 hours to get bare-trusty nodes in the check pipeline now22:22
jeblair2014-09-03 18:50:01.099 | 441 bashate error(s) found22:22
jeblairin config22:22
*** packet has joined #openstack-infra22:22
clarkbjeblair: I am rereading the filter code in and no obvious subprocess pipe leaks stand out. We may want to put lines 169-170 in something more like a finally block though22:24
clarkbjeblair: does anything stand out to you?22:24
clarkbI am going to dig into logs further back in time to see if there is an error thrown before we start leaking22:25
arosenI've been running into this probably lately i'm not sure if i'm doing something wrong... When I do git review or git push manually i get back:
arosenit only happens when i have multiple patches rebased on each other22:26
*** dimtruck is now known as zz_dimtruck22:26
jeblairclarkb: yeah, a try/finally sounds good.  an error on f.close() could prevent other filters folm closing (i'm not sure if there is more than one filter).  logq.put could fail if the queue is full; i don't know if that happens either.22:27
fungiarosen: usually that means that you updated the commit without changing its parentage or modifying its content or commit message22:27
*** kurtmartin has quit IRC22:27
*** andreykurilin_ has quit IRC22:28
arosenfungi:  right but if i do git commit --amend and change the top commit message I should be able to repush even if the patches below haven't changed right?22:28
*** annegent_ has quit IRC22:28
arosenIv'e never really had this problem in the past :/22:28
fungiarosen: right22:28
fungiarosen: how are you building the patch series? git review -d?22:29
clarkbjeblair: I think logq.put() should block if the queue is full22:29
*** jgrimm is now known as zz_jgrimm22:29
fungi(to retrieve the existing series from gerrit into your locak working clone i mean)22:29
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/publications: Update slides for Fossetcon
*** kurtmartin has joined #openstack-infra22:30
pleia2bah, how did I manage to push that to template22:31
pleia2I clearly don't do this often enough :)22:31
fungipleia2: if you branched from template, you need to update the defaultbranch in .gitreview on the new branch to help you not do that22:31
arosenfungi: I'm not sure the work flow that breaks it. git add file; git commit; git review; git add file; git commit; git review;22:31
pleia2ah, I never updated .gitreview in this branch22:32
pleia2fungi: yeah :)22:32
clarkbjeblair: we should probably also close stdout and stderr in I will write a patch for these things22:32
openstackgerritElizabeth K. Joseph proposed a change to openstack-infra/publications: Update slides for Fossetcon
fungiarosen: if you have an example you hit, being able to see the contents of git log --online would help22:33
fungier, --oneline22:33
openstackgerritA change was merged to openstack-infra/config: Remove Octavia python 3.3 job, as eventlet means that will always fail.
openstackgerritA change was merged to openstack-infra/config: Update documented local gerrit git replica path
*** marcoemorais has quit IRC22:33
*** marcoemorais has joined #openstack-infra22:34
arosenfungi: see i changed the comment messsage to !2222:34
arosene2490d5 --> isn't showing up in git log --online which is weird.22:34
fungiarosen: yeah, the issue is back at the "Fixup gitignore" change22:35
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add the tempest-lib project to openstack
fungiarosen: apparently it is an update of but not significantly different than the version in gerrit of
arosenfungi: but the patches on top of that changed though22:36
arosenso i should be able to push the new stuff.22:36
fungiarosen: yeah, but that commit changed too, and gerrit doesn't like that it's not got the same gitsha as
*** sdake_ has quit IRC22:37
arosenfungi: Ah I see: so I did a git rebase -i HEAD~5; then changed the order of commits so the git sha changes22:38
fungiarosen: i would do 'git review -d 118657;git cherry-pick 4602d2d;git cherry-pick 271f8db;git cherry-pick bc5383e;git review'22:38
*** imeerovi has quit IRC22:38
*** krtaylor has joined #openstack-infra22:38
fungiarosen: except when you were reordering, i think you went one commit too far back and re-picked e2490d5 which when it was amended got a new sha of f6b0b73 but was otherwise unchanged (same parent gitsha, same commit message, same diff)22:40
*** bknudson has quit IRC22:41
fungiso gerrit sees you trying to push f6b0b73 as a new patchset to with zero delta22:41
*** emagana has joined #openstack-infra22:42
arosenfungi:  ah i see. Thanks for explaining that to me. That's exactly what happened.22:43
*** masayukig has quit IRC22:44
*** otherwiseguy has quit IRC22:45
*** craigbr has quit IRC22:50
*** imeerovi has joined #openstack-infra22:50
*** masayukig has joined #openstack-infra22:50
*** arnaud has quit IRC22:51
*** pcm_ has quit IRC22:51
*** dims has quit IRC22:52
fungino meetings at the moment. approving 11652722:52
*** dims has joined #openstack-infra22:52
fungiwell, i'll wait until 23:05 in case someone starts one at 23:00 (given the current gating sluggishness)22:52
*** craigbr has joined #openstack-infra22:53
*** marun_ has quit IRC22:54
*** dims has quit IRC22:57
*** gyee has quit IRC22:59
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Adds Ironic parallel Tempest job
*** paulrad has quit IRC22:59
*** paulrad has joined #openstack-infra22:59
openstackgerritEdward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally
*** Sukhdev has quit IRC23:00
fungihuh... what was the deal with bashate running against openstack-infra/config?23:02
openstackgerritAaron Rosen proposed a change to openstack-dev/cookiecutter: Add additional gitignores .swo/.swn
jeblairfungi: that doesn't imapct eavesdrop23:03
fungiit looks like it's failing to exclude the .tox directory, and is descending into its own sample failure tests23:03
fungiRunning bashate on ./.tox/bashate/lib/python2.7/site-packages/bashate/tests/samples/E001_bad.sh23:03
fungiet cetera23:03
*** paulrad has quit IRC23:04
anteayamorning jhesketh23:05
sdaguefungi: yeh, there is a thread on trying to figure out the ignore parts of this correctly23:06
sdagueon the -dev ML23:06
fungijeblair: oh, ha yeah... i was thrown off by clarkb's comment that he wasn't approving the change because there were meetings going on. forgetting that the change wasn't touching meetbot configuration23:06
*** jergerber has quit IRC23:06
*** masayukig has quit IRC23:07
openstackgerritClark Boylan proposed a change to openstack-infra/config: Handle log processing subprocess cleanup better
clarkbfungi: oh it didnt?23:07
clarkbI thought  Iexpanded it and it did23:07
clarkbjeblair: ^ how does that look to you? I haven't tested it but I figured early feedback would be good23:08
jeblairfungi: i was too, but then i went to try to figure out why i missed it in my search of eavesdrop changes earlier :)23:08
fungijeblair: i had no excuse... it was my patch after all23:09
clarkbjeblair: secifically do we need to read() the pipes before closing them?23:09
clarkbjeblair: close(2) shouldn't error if there is data in the pipe right?23:09
*** wenlock has quit IRC23:09
*** masayukig has joined #openstack-infra23:09
*** SumitNaiksatam has quit IRC23:10
*** signed8b_ has quit IRC23:11
clarkboh that was a statusbot list to eavesdrop23:11
clarkbI saw eavesdrop and immediately thought: meetbot23:11
jeblairclarkb: erm, i don't really like the "always abruptly terminate" approach23:12
clarkbjeblair: without it we can hang on wait()23:13
jeblairclarkb: is that happening?23:13
clarkbjeblair: I haven't observed it23:13
openstackgerritA change was merged to openstack-infra/config: Add #openstack-stable IRC channel
clarkbjeblair: but with my change to not read the pipe until it is empty it may hang?23:14
jeblairclarkb: i dunno.  that's just a lot of change without actually having an error that we're trying to fix23:14
*** david-lyle has quit IRC23:14
clarkbjeblair: it was to make the close() calls simpler23:14
clarkbas crm114 may not die if I just close my side of the pipe23:15
clarkbI may have to read all the data for it to EOF and shutdown nicely?23:15
jeblairclarkb: i'm just saying i do not understand the problem that's trying to be fixed, and can't determine whether your change addresses it23:15
clarkbjeblair: the problem is the Popen is leaky23:15
jeblairclarkb: the symptom is that pipes have leaked23:16
jeblairclarkb: i helped you speculate as to possible causes, but unless i missed something, i don't think you identified one23:16
*** signed8bit has joined #openstack-infra23:17
clarkbjeblair: I did not pdb/strace it to find where the leak is happening23:18
clarkbbut there are several code paths that may throw short circuiting clean up23:18
jeblairclarkb: i agree; but you're doing more than that here, you're changing the way it's closed23:18
clarkbjeblair: yes, to simplify the closure23:19
*** openstackstatus has quit IRC23:19
jeblairclarkb: i don't know why, nor do i know what the implications are for that.  for both this program, and the program you are terminating23:19
jeblairclarkb: particularly, i can't answer the question "why should we never let it terminate normally?"23:19
clarkbjeblair: if I remove the kill() then I think I have to do crazy non blocking reads off of the pipes until they are empty23:20
*** otherwiseguy has joined #openstack-infra23:20
*** openstackstatus has joined #openstack-infra23:20
*** ChanServ sets mode: +v openstackstatus23:20
*** dane_leblanc has quit IRC23:21
jeblairclarkb: does that increase the risk of database corruption?23:21
clarkbjeblair: if the reads are not non blocking the parent process may deadlock the subprocess and if  Idon't read all of the data to completion the subprocess may not die blocking the parent on wait(). That said the subprocess may not take kindly to being killed23:21
*** Sincler has quit IRC23:22
*** ianw has quit IRC23:25
*** ianw has joined #openstack-infra23:25
NobodyCamHi -infra : off the wall question: is there a way to recheck a reuuning job with out pushing up a new patch?23:25
*** dmsimard is now known as dmsimard_away23:26
*** asettle has quit IRC23:26
openstackgerritA change was merged to openstack-infra/config: Fixing deprecation warnings in logrotate module
anteayaNobodyCam: not that I know of23:28
anteayaNobodyCam: using recheck will just wait until the current tests finish, as far as I know23:28
anteayaNobodyCam: why do you ask?23:28
clarkbjeblair: I can stab at using select and removing the kill23:28
NobodyCamI have a job that failed py27 due to bug and it like 4 hours away form finishing23:29
*** baoli has joined #openstack-infra23:29
*** coolsvap has quit IRC23:29
*** chuckC has quit IRC23:30
anteayaNobodyCam: you aren't approved, change the commit message23:30
*** emagana has quit IRC23:31
NobodyCamoh good idea anteaya TY23:31
*** emagana has joined #openstack-infra23:32
clarkbjeblair: in the mean time what do you think about splitting the cahnge into and merging the try finally in the outer block more quickly?23:32
anteayaNobodyCam: it used to work, let's see if it still does23:32
clarkbjeblair: then we can sort out how the inner close() works in a second patch23:32
fungiNobodyCam: changing the commit message will indeed restart testing, but it will also dump any existing reviewer votes since it effectively creates a new patchset for you23:33
*** coolsvap has joined #openstack-infra23:33
NobodyCamAwesome TY :)23:33
*** hogepodge has quit IRC23:34
*** atiwari has quit IRC23:34
*** _buzz_ has quit IRC23:35
*** emagana_ has joined #openstack-infra23:36
*** hemna is now known as hemna__23:36
*** emagana has quit IRC23:36
*** alexpilotti has quit IRC23:37
*** dangers is now known as dangers_away23:37
*** _buzz has joined #openstack-infra23:38
jrollclarkb: I learned about new things today: "the hypervisor and xapi have lost connection in such a way where the HV cannot allocate new instance chunks or make certain changes"23:40
*** koolhead17 has joined #openstack-infra23:40
clarkbjroll: all I hear is "Something xen xen something something xen unhappy xen" :)23:41
jrollclarkb: right, exactly :P23:41
jrollbut yeah, memory issue23:41
jrollthanks again for noticing that :)23:41
*** _buzz has quit IRC23:42
*** otherwiseguy has quit IRC23:43
*** otherwiseguy has joined #openstack-infra23:43
*** _buzz has joined #openstack-infra23:44
clarkbjeblair: as an alternative it may be time to look into using crm114 in daemon/service mode23:44
clarkbjeblair: then we can launch it independently23:44
clarkband not try to do process management in these small scripts23:45
*** mwagner_lap has joined #openstack-infra23:45
openstackgerritK Jonathan Harker proposed a change to openstack-infra/config: Reduce graphite's stats retention policy
*** _buzz has quit IRC23:49
*** _buzz has joined #openstack-infra23:49
*** _buzz has quit IRC23:50
*** gokrokve has quit IRC23:50
fungibug 1365139 looks like fun, but i don't have the remaining energy tonight to wontfix it and explain why it's "not going to happen[tm]"23:50
uvirtbotLaunchpad bug 1365139 in openstack-ci "partial recheck" [Undecided,New]
*** reed has quit IRC23:53
clarkbjeblair: a quick test indicates closing stdin is enough to kill crm23:54
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Removed sort field from search.
*** kurtmartin has quit IRC23:55
*** baoli has quit IRC23:56
*** baoli has joined #openstack-infra23:56
*** _buzz has joined #openstack-infra23:57
*** mbacchi has quit IRC23:58
*** chuckC has joined #openstack-infra23:58

Generated by 2.14.0 by Marius Gedminas - find it at!