Wednesday, 2016-09-28

clarkbcloudnull: assuming we have enough load you should be able to see those new flavors get used in the mean time00:00
mtreinishclarkb, fungi: https://github.com/eclipse/mosquitto/issues/27800:00
fungiokay, logstash.o.o is online again00:00
cloudnullclarkb: it looks like an s3500 node is in process now :)00:00
fungimtreinish: that's a beaut00:00
cloudnullso it "should"-tm be working00:01
cloudnulli'll be keeping an eye on things as it gets spinning00:01
*** aeng__ has joined #openstack-infra00:01
clarkbcloudnull: sounds good00:01
cloudnullthanks again eveyone for all the assistance.00:02
*** aeng_ has quit IRC00:02
fungiclarkb: tailing /var/log/logprocessor/log-client-debug.log and so far just one line: 2016-09-28 00:00:29,813 Log pusher starting.00:02
fungiso it's not like it's going crazy at the moment at least00:02
*** thcipriani is now known as thcipriani|afk00:02
*** roxanaghe has quit IRC00:02
*** thorst has joined #openstack-infra00:03
clarkbalso some deduping of images did happen despite the races00:03
clarkbfungi: you should see things generated theer when jobs start finishing00:03
*** spzala_ has quit IRC00:04
*** claudiub|2 has quit IRC00:04
*** mriedem has joined #openstack-infra00:04
fungiyeah, i expect just no jobs have completed yet00:04
clarkbdoesn't look like the process is stillrunning00:04
*** sflanigan_ has quit IRC00:04
clarkboh!00:04
fungioh, huh...00:04
fungistale pidfile?00:04
clarkbwe might still be affected by the thing where dns isn't working when the service starts so it dies00:04
clarkbbut maybe also a stale pid file00:05
fungiaha00:05
fungii'll start it manually and make sure it keeps going00:05
clarkbupstart + sysv init scripts makes this very hard00:05
clarkbfungi: ok00:05
*** pahuang__ has quit IRC00:05
jlvillalAny chance we can get a review on https://review.openstack.org/#/c/352909/  It already has two +2 votes.  This is a devstack-gate patch. Trying to get Ironic multi-node testing going.00:05
clarkbI should just make systemd unit files for them then we can upgrade them to xenial real soon once precise is gone00:05
*** thorst has quit IRC00:05
anteayafungi: thanks for reapplying your +1 to 376828 after I rebased, I appreciate it00:05
fungiclarkb: yeah, i found no stale pidfile anyway00:05
*** xarses has quit IRC00:06
*** aeng__ has quit IRC00:06
fungithe log is definitely active now00:06
fungi_very_ active00:06
*** zhurong has quit IRC00:06
fungi200k lines of active just in the first minute since starting it00:07
fungimaybe this is a little _too_ debug?00:07
clarkbya we might want to consider pulling back on how much we log there00:07
clarkbthough I don't know why it skyrocketed in july00:07
clarkboh maybe that was when we put zuul launchers in place?00:08
fungioh yeah, almost up to 50mib already00:08
clarkband that is basically directly correlated to traffic from the job launchers whether zuul or jenkins00:08
clarkbjlvillal: done00:09
jlvillalclarkb: Thank you :)00:09
clarkbjlvillal: out of curiousity was that causing failure because c-vol/c-bak fail to start if no c-api is running?00:10
clarkb(I always thought our services were supposed to be resilient to that...)00:10
jlvillalclarkb: I believe so. On the Ironic testing we don't have any cinder at all.00:10
jlvillalclarkb: So when the subnode would try to start it, it would fail.00:11
*** davidlenwell has quit IRC00:11
*** edmondsw has quit IRC00:11
fungiclarkb: so with zuul mostly idle at the moment, the log client is recording an average of 300kib of debug logs a second. if my math is correct, it will fill up the rootfs again at this rate in 34 hours time00:12
clarkbfungi: fun00:13
*** amitgandhinz has joined #openstack-infra00:13
*** ddieterly has quit IRC00:13
*** ijw has quit IRC00:14
*** tpsilva has quit IRC00:14
clarkbfungi: quick interim fix would be to remove the -d/--debuglog from the command invocation00:14
*** kaisers2 has joined #openstack-infra00:15
clarkbfungi: I can push taht patch in a sec00:15
fungiahh, thanks. starting to run out of steam over here00:16
*** kaisers3 has joined #openstack-infra00:16
*** kaisers has quit IRC00:17
*** kaisers1 has quit IRC00:17
*** aeng__ has joined #openstack-infra00:17
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Reduce log client logging by default  https://review.openstack.org/37810400:18
clarkbfungi: ^ that should do it00:18
clarkbthough we should write a better long term logging setup for it00:18
*** ijw has joined #openstack-infra00:19
*** sflanigan_ has joined #openstack-infra00:19
clarkbcloudnull: ok all iamges are uploaded to osic's logical providers with some deduping too00:19
*** markvoelker has joined #openstack-infra00:20
cloudnullso the image upload skip bits are working as expected?00:20
clarkbfor the most part00:21
cloudnullthats great !00:21
clarkbits not 100% since they can race each other but where the racing happens the way we want it looks like it works00:21
cloudnullI see the new images look like "nodepool-ubuntu-xenial-1160"00:21
*** pahuang__ has joined #openstack-infra00:21
cloudnulland the old ones were "template-debian-jessie-1474978264"00:21
cloudnullis that right ?00:22
clarkbcloudnull: yup00:22
cloudnullthen it's lookin good on my end too.00:22
cloudnull:)00:22
clarkbcloudnull: also left a comment on your new change. its not quite right00:22
cloudnullok00:22
* cloudnull goes to look00:22
clarkbcloudnull: basically we have to leave the old provider in place with a max server value of inactive00:22
clarkbso that nodepool can gracefully clean up the resources it associates with that provider00:22
clarkbjeblair: when you return tomorrow I also think we might have a race on the delete side ... that one could get a bit nasty00:23
clarkbjeblair: basically with this new dedup stuff we want to treat deletes on the whole of the deduped images00:23
clarkbotherwise we could remove the image from one provider and not realize another provider is using it and leads to boot fails00:23
clarkbthis can happen if the upload time between providers is long (it shouldn't be so likely won't be a problem in practice)00:24
* clarkb likes problems that are not problems in practice00:24
clarkblike the added cancer risk of going to mars according to elon musk00:24
clarkb"it will be higher but not that much higher"00:24
cloudnullso should I set it to "inactive" or 0?00:25
*** benk01 has joined #openstack-infra00:25
clarkbcloudnull: to -100:25
benk01Hello.00:25
*** markvoelker has quit IRC00:25
clarkbmax-servers: -100:25
*** davidlenwell has joined #openstack-infra00:25
clarkbbenk01: hello00:25
cloudnulland should I leave the image entries in this case ?00:25
cloudnullthe -name: ...\n... - name: osic-cloud1 ?00:26
clarkbcloudnull: under labels you mean? i think you can continue to remove them but its not necessary unless we completely remove the provider later00:26
cloudnullok00:27
benk01Hey guys, my review.openstack.org account is in a state of flux. I ran through the "Agreementes" section; though left my address details bare. It seems these details are needed in order to submit commits to gerrit and when I try and enter such details I get "server error: cannot store contact information"00:28
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Update server distribution to HW specific entries  https://review.openstack.org/37810000:28
benk01like; I have done my ICLA, though I was prompted later to fill in contact info.00:28
clarkbyes in order to properly sign the cla you need to include contact info00:28
*** xarses has joined #openstack-infra00:29
cloudnullso i put it all back and just changed the max-server settings. If we need to remove the provider all together I'll cross that bridge when I get there.00:29
clarkbcloudnull: ok00:30
*** adrian_otto has quit IRC00:30
cloudnullunless you think its best to remove those labels now .00:30
benk01clarkb: how do I complete the ICLA/contact information storage? review.openstack keeps throwing errors when I try to store it.00:30
fungibenk01: your foundation profile primary e-mail address at www.openstack.org needs to match your preferred e-mail address in gerrit's contact settings00:31
*** tqtran has quit IRC00:31
fungithe contact storing form in gerrit is an api callback to the foundation member system to update the linkage against your foundation profile00:31
clarkbcloudnull: nah its fine to deal with it when we get there00:32
cloudnullok00:32
benk01oh ok. I don't think I have a foundation profile yet. I will create one now.00:32
*** markvoelker has joined #openstack-infra00:34
kzaitsev_mbwhat should I do if I have a project in glaobl-requirements and all the jenkins jobs fail with Could not satisfy constraints for 'murano-pkg-check': installation from path or url cannot be constrained to a version00:35
kzaitsev_mbthere must be a workaround that is used by clients and oslo libs, right?00:35
*** gordc has joined #openstack-infra00:35
cloudnullclarkb: it looks like image 9b34985b-a92e-42cb-8dbe-29d39f54c06d "nodepool-ubuntu-xenial-1160" is queued and has been for a while. maybe the upload didn't complete there?00:36
clarkbhrm nodepool thinks they all finished, I winder if that got lost in my restart of the builder00:39
fungibenk01: http://docs.openstack.org/infra/manual/developers.html#account-setup is what we normally expect new users to find (it's linked from lots of places)00:39
clarkbcloudnull: does it look like it is hurting anything? if not I think I will debug in the morning00:39
cloudnullnope its now00:39
clarkbso I canstart on evening things now00:39
cloudnull*not00:39
cloudnullso cheers, and have a good night !00:40
clarkbyou too00:40
*** caowei has joined #openstack-infra00:40
*** Douhet has joined #openstack-infra00:42
*** doffm has quit IRC00:42
*** adrian_otto has joined #openstack-infra00:42
*** doffm has joined #openstack-infra00:42
*** gyee has quit IRC00:43
*** nijaba has quit IRC00:44
*** ijw has quit IRC00:45
*** Apoorva_ has joined #openstack-infra00:45
*** nijaba has joined #openstack-infra00:45
*** nijaba has joined #openstack-infra00:45
*** bin_ has quit IRC00:46
*** ddieterly has joined #openstack-infra00:46
*** bin_ has joined #openstack-infra00:46
*** adrian_otto has quit IRC00:46
*** adrian_otto has joined #openstack-infra00:47
*** psilvad has quit IRC00:47
benk01Thanks; I think my account is now set up.00:47
*** amitgandhinz has quit IRC00:47
*** salv-orlando has joined #openstack-infra00:48
*** Apoorva has quit IRC00:48
*** Apoorva_ has quit IRC00:50
*** tuannguyen has joined #openstack-infra00:50
*** shu-mutou-AWAY is now known as shu-mutou00:52
*** salv-orlando has quit IRC00:52
*** tuannguyen has quit IRC00:54
*** M-docaedo_vector has joined #openstack-infra00:58
*** ddieterly has quit IRC01:01
dimsanyone with git admin privs around? i typo'ed a branch creation01:02
*** kvcobb has quit IRC01:04
fungii can clean it up. what do you need?01:04
*** benk01 has quit IRC01:04
*** Goneri has quit IRC01:04
fungidims: ^01:05
dimsah, one sec01:05
*** ijw has joined #openstack-infra01:05
dimsfungi : need to nuke http://git.openstack.org/cgit/openstack/tacker/log/?h=stable/newton, i typoed the branch creation of a mitaka SHA01:06
armaxhi folks sorry for the (silly) question01:06
armaxbut how does one retire a project?01:06
*** adrian_otto1 has joined #openstack-infra01:06
dimsfungi : thanks a ton!01:06
dimsarmax : http://ronaldbradford.com/blog/retiring-an-openstack-project-2016-03-28/01:06
armaxhttp://docs.openstack.org/infra/manual/drivers.html#retiring-a-project?01:06
fungiarmax: http://docs.openstack.org/infra/manual/drivers.html#retiring-a-project01:06
dimsarmax : y that blog has a link to http://docs.openstack.org/infra/manual/drivers.html#retiring-a-project01:06
armaxdims: ack01:07
armaxI am thinking of retiring neutron01:07
armaxthanks dims :)01:07
fungiarmax: you're going to get some people excited if you say that ;)01:07
dimsarmax : haha back to nova-network? :)01:07
armaxno, no networking at all01:08
armaxnetworking is overrated anyway01:08
* dims hands out pitchforks 01:08
*** adrian_otto has quit IRC01:08
dimsarmax : ooh, i neeed your +1 in https://review.openstack.org/#/c/374820/01:09
armaxdims: sure thing01:09
dimsthank you sir!01:09
armaxlooking01:09
harlowjano more networking ++01:09
openstackgerritMerged openstack-infra/devstack-gate: Add c-vol,c-bak on subnode when c-api enabled  https://review.openstack.org/35290901:10
fungidims: i can't delete it--there are open changes for that branch that need to get merged or abandoned first01:10
*** rvba` has quit IRC01:10
*** ijw_ has joined #openstack-infra01:10
dimsone sec let me see if i can find those folks01:10
fungiarmax: maybe we can use cinder to tunnel ip over fiber channel? (like a reverse of iscsi?)01:10
*** senk_ has joined #openstack-infra01:11
armaxfungi: I am sure that’s a lot less scary than what neutron has become01:11
harlowjapigeons01:11
*** ijw has quit IRC01:11
dimsfungi : clean now https://review.openstack.org/#/q/project:openstack/tacker+branch:stable/newton01:11
dimsit was just my .gitreview change01:11
fungidims: i've deleted stable/newton for openstack/tacker, formerly at 04a973be4e5106c0bf001de9054a46a4d7a8846301:12
*** rvba has joined #openstack-infra01:12
*** rvba has quit IRC01:12
*** rvba has joined #openstack-infra01:12
dimsharlowja : while we are at it nova too? (see https://github.com/Mirantis/virtlet)01:12
dimsfungi : thanks a ton for the quick response!01:12
harlowjadims hmmm, i was wondering when something like that would appear01:12
dimsarmax harlowja - am kidding of course01:12
harlowjaha01:12
*** kaisers__ has quit IRC01:12
harlowjaha01:12
harlowja:-/01:12
dimsharlowja : it's a experiment :)01:12
harlowja:)01:13
harlowjaya, its hard for me to decide if k8s is just openstack at a much earlier phase in its life (except with containers!)01:13
harlowjaand with a slightly bigger backer01:14
fungicontainers and pixy dust01:14
*** gildub has quit IRC01:14
harlowjayup01:14
*** gongysh has joined #openstack-infra01:14
harlowjabut dims ya i expect virtlet or something like it soonish01:14
harlowjaits a natural question of 'why just containers'01:14
*** senk_ has quit IRC01:15
dimsharlowja : fungi : right01:15
*** ijw_ has quit IRC01:15
harlowjalife is like a big circle01:16
harlowjalol01:16
* fungi thinks time sharing systems are making a comeback01:16
dims:)01:16
harlowjanot yet, but soon, lol01:17
harlowjaout for the day, peace01:17
dimsgood evening harlowja01:18
*** zhurong has joined #openstack-infra01:18
*** rossella_ has quit IRC01:19
*** gordc has quit IRC01:19
*** rossella_ has joined #openstack-infra01:19
*** gildub has joined #openstack-infra01:20
dimsarmax : thanks for the +1, agree on the add/revert of the workaround01:21
armaxack01:22
*** lezbar has quit IRC01:22
*** lezbar has joined #openstack-infra01:23
*** pkoniszewski has quit IRC01:25
*** pkoniszewski has joined #openstack-infra01:31
*** thorst has joined #openstack-infra01:32
*** thorst has quit IRC01:32
*** yanyanhu_ has joined #openstack-infra01:38
*** lezbar has quit IRC01:40
*** lezbar has joined #openstack-infra01:40
*** amitgandhinz has joined #openstack-infra01:44
dimsfungi :  please nuke http://git.openstack.org/cgit/openstack/tacker-horizon/log/?h=stable/newton as well. that's in the same typo-ed batch01:46
*** tonytan4ever has joined #openstack-infra01:46
*** rushil has joined #openstack-infra01:47
*** ddieterly has joined #openstack-infra01:49
*** tonytan4ever has quit IRC01:50
*** kzaitsev_mb has quit IRC01:51
*** ddieterly has quit IRC01:52
*** hongbin has joined #openstack-infra01:54
*** gema has quit IRC01:56
openstackgerritzhangyanxian proposed openstack-infra/project-config: Fix typo in openstack_functions.py  https://review.openstack.org/37814301:57
*** Jeffrey4l has joined #openstack-infra01:58
*** kmartin has quit IRC02:02
*** gema has joined #openstack-infra02:03
*** liusheng has quit IRC02:03
openstackgerritzhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project zaqar  https://review.openstack.org/37814302:04
*** liusheng has joined #openstack-infra02:04
*** tphummel has quit IRC02:04
*** baoli has quit IRC02:05
*** baoli has joined #openstack-infra02:11
*** tuannguyen has joined #openstack-infra02:12
*** amitgandhinz has quit IRC02:18
*** ijw has joined #openstack-infra02:19
*** ddieterly has joined #openstack-infra02:21
*** ijw has quit IRC02:23
*** ijw has joined #openstack-infra02:24
*** rlandy has quit IRC02:24
*** ddieterly has quit IRC02:25
*** adrian_otto1 has quit IRC02:25
*** sflanigan_ has quit IRC02:26
*** ijw has quit IRC02:29
*** mriedem has quit IRC02:30
*** ijw has joined #openstack-infra02:31
*** salv-orlando has joined #openstack-infra02:36
*** sflanigan_ has joined #openstack-infra02:39
*** salv-orlando has quit IRC02:40
*** shu-mutou has quit IRC02:44
*** ijw has quit IRC02:44
*** shu-mutou has joined #openstack-infra02:44
*** baoli has quit IRC02:52
*** lezbar has quit IRC02:56
*** lezbar has joined #openstack-infra02:57
*** Sukhdev has joined #openstack-infra02:59
*** tuanla has joined #openstack-infra03:00
*** Sukhdev has quit IRC03:01
*** david-lyle has quit IRC03:04
*** spzala has joined #openstack-infra03:05
*** spzala has quit IRC03:05
*** hongbin has quit IRC03:10
*** adrian_otto has joined #openstack-infra03:12
*** rfolco_ has quit IRC03:14
*** amitgandhinz has joined #openstack-infra03:14
*** sdake_ has quit IRC03:17
*** armax has quit IRC03:18
openstackgerritCao Xuan Hoang proposed openstack/os-client-config: Using assertIsNone() instead of assertEqual(None, ...)  https://review.openstack.org/37820503:25
*** adrian_otto has quit IRC03:26
*** jamesden_ has quit IRC03:26
*** adrian_otto has joined #openstack-infra03:26
*** vikrant has joined #openstack-infra03:26
*** tuannguyen has quit IRC03:29
*** tqtran has joined #openstack-infra03:30
*** rajinir has quit IRC03:35
*** tqtran has quit IRC03:36
*** roxanaghe has joined #openstack-infra03:38
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Updated grafana to support new OSIC HW specs  https://review.openstack.org/37821503:38
*** sdake has joined #openstack-infra03:39
cloudnullSo in osic-cloud1 it looks like we're missing the xenial image from tonights upload. I know folks are likely AFK now but when possible if we can get that image re-uploaded I think it'll make our error node count decline.03:44
irtermite:(03:44
cloudnullI don't know what's specifically causing http://grafana.openstack.org/dashboard/db/nodepool-osic to show error node launch attempts but i guessing it's because of that image missing.03:45
*** amitgandhinz has quit IRC03:49
*** sdake_ has joined #openstack-infra03:50
*** Sukhdev has joined #openstack-infra03:50
*** sdake has quit IRC03:51
*** roxanaghe has quit IRC03:52
*** adrian_otto has quit IRC03:53
*** tongli_ has quit IRC03:57
*** gongysh has quit IRC03:58
*** roxanaghe has joined #openstack-infra03:59
*** njohnston__ is now known as njohnston04:01
*** ramishra has quit IRC04:03
*** sflanigan_ has quit IRC04:04
*** rkukura has quit IRC04:05
*** ramishra has joined #openstack-infra04:07
*** yamamoto_ has quit IRC04:17
*** sflanigan_ has joined #openstack-infra04:20
*** pabelanger has quit IRC04:22
*** sai has quit IRC04:22
*** roxanaghe has quit IRC04:23
*** sai has joined #openstack-infra04:23
*** pabelanger has joined #openstack-infra04:23
*** amotoki has joined #openstack-infra04:28
*** claudiub|2 has joined #openstack-infra04:31
*** Rocky_g has quit IRC04:36
*** psachin has joined #openstack-infra04:41
*** amitgandhinz has joined #openstack-infra04:45
*** gongysh has joined #openstack-infra04:49
*** yamamoto has joined #openstack-infra04:59
*** rkukura has joined #openstack-infra05:00
*** pgadiya has joined #openstack-infra05:07
*** Sukhdev has quit IRC05:08
*** links has joined #openstack-infra05:09
*** links has quit IRC05:15
*** links has joined #openstack-infra05:17
*** salv-orlando has joined #openstack-infra05:18
*** rossella_ has quit IRC05:19
*** rossella_ has joined #openstack-infra05:19
*** amitgandhinz has quit IRC05:20
*** sshnaidm|afk is now known as sshnaidm05:21
*** salv-orlando has quit IRC05:22
*** prometheanfire has quit IRC05:23
*** gildub has quit IRC05:26
*** gildub has joined #openstack-infra05:28
*** nadya has joined #openstack-infra05:29
*** woodster_ has quit IRC05:30
*** prometheanfire has joined #openstack-infra05:30
*** sdake_ has quit IRC05:30
*** nadya has quit IRC05:31
*** flepied has quit IRC05:32
*** ssancheztrujillo has joined #openstack-infra05:32
*** apetrich has quit IRC05:32
*** apetrich has joined #openstack-infra05:37
*** camunoz_ has joined #openstack-infra05:38
*** kaisers has joined #openstack-infra05:39
*** jaosorior has joined #openstack-infra05:46
*** nherciu has joined #openstack-infra05:53
*** j_king has quit IRC05:53
*** rushil has quit IRC05:59
*** yuval has joined #openstack-infra06:00
ramishraHi guys, we're seeing numerous dsvm gate job failures with error connecting to the fedora mirrors (ex.  Failed to connect to mirrors.mit.edu port 80: Connection timed out). Are there any infra network issues atm?06:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix upstream tracking repos  https://review.openstack.org/37801606:08
*** rcernin has joined #openstack-infra06:14
*** nadya has joined #openstack-infra06:14
*** amitgandhinz has joined #openstack-infra06:16
*** nadya has quit IRC06:18
*** andreas_s has joined #openstack-infra06:24
*** danielitit has joined #openstack-infra06:27
*** aeng__ has quit IRC06:30
*** aeng__ has joined #openstack-infra06:30
*** crinkle_ is now known as crinkle06:33
*** sflanigan_ has quit IRC06:34
openstackgerritRobin Naundorf proposed openstack-infra/system-config: Fix small typos in docs  https://review.openstack.org/37769106:35
*** mrunge has quit IRC06:36
*** andreas_s has quit IRC06:36
*** Ravikiran_K has joined #openstack-infra06:36
*** pcaruana has joined #openstack-infra06:37
*** flepied has joined #openstack-infra06:37
*** ihrachys has joined #openstack-infra06:37
*** andreas_s has joined #openstack-infra06:38
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Enable layout-checks.py again  https://review.openstack.org/37827506:39
*** ihrachys has quit IRC06:40
*** ihrachys has joined #openstack-infra06:41
*** mrunge has joined #openstack-infra06:42
openstackgerritRobin Naundorf proposed openstack-infra/system-config: Fix deadlink in wiki doc  https://review.openstack.org/37828006:43
*** ihrachys has quit IRC06:44
*** sflanigan_ has joined #openstack-infra06:45
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: make upgrade jobs voting  https://review.openstack.org/37828406:48
AJaegerproject-config cores, could you review two changes to fix track-upstream repos, please? https://review.openstack.org/378003 and https://review.openstack.org/37801606:49
*** amitgandhinz has quit IRC06:50
*** salv-orlando has joined #openstack-infra06:50
*** drifterza has joined #openstack-infra06:54
*** isaacb has joined #openstack-infra06:55
*** senk_ has joined #openstack-infra06:56
*** pahuang__ has quit IRC06:56
senk_i guess something is not working on this one https://review.openstack.org/#/c/329410/06:57
AJaegersenk_: see "Related changes" - it's dependent on two other changes. Those need to merge first.06:58
senk_ah okay thank you for pointing me to that AJaeger :)06:58
*** muawiakhan has joined #openstack-infra06:59
*** tuanla has quit IRC07:01
*** [HeOS] has joined #openstack-infra07:02
*** gildub has quit IRC07:04
*** amoralej|off is now known as amoralej07:04
yolandagood morning07:05
*** ralonsoh has joined #openstack-infra07:06
therveAJaeger, we're getting awful external connectivity in the gate in the past 2 days, I don't know if there is something going on07:08
*** pahuang__ has joined #openstack-infra07:08
AJaegermorning, yolanda !07:09
*** gildub has joined #openstack-infra07:09
therveMostly OSIC cloud apparently, though it seems to be running most of our jobs07:09
AJaegertherve: that's sad - could you summarize on openstack-infra, please? This needs cloudnull and the US based infra-roots.07:09
*** lezbar has quit IRC07:10
AJaegertherve: there's some work on changing OSIC setup going on, but reference to a few jobs/log files that show the behaviour would be great07:10
*** matrohon has joined #openstack-infra07:10
therveAJaeger, WDYM my openstack-infra?07:10
*** lezbar has joined #openstack-infra07:10
AJaegertherve: http://lists.openstack.org/pipermail/openstack-infra/07:11
therveOK sure07:11
AJaegertherve: I should have mentioned *mailing list*07:11
therveNo worries :)07:11
AJaegeryolanda: when you review, please prioritize https://review.openstack.org/377738 https://review.openstack.org/377947  https://review.openstack.org/378003 and https://review.openstack.org/37801607:12
*** links has quit IRC07:16
yolandahi sure07:17
AJaegerthanks, yolanda07:24
*** camunoz_ has quit IRC07:28
openstackgerritMerged openstack-infra/project-config: Move periodic-python-jobs-with-neutron-lib-master to xenial  https://review.openstack.org/37773807:31
*** jlanoux has joined #openstack-infra07:33
*** winggundamth has joined #openstack-infra07:33
*** tuannguyen has joined #openstack-infra07:33
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: test upgrade from newton branch  https://review.openstack.org/37830507:34
*** nmagnezi has joined #openstack-infra07:35
*** jpich has joined #openstack-infra07:35
*** tuannguyen has quit IRC07:39
*** jpena|off is now known as jpena07:40
AJaegerdims, http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/32d4c97/console.html is failing every time - looks like the tox.ini generated does not work. Could you check and fix it? Or should we remove it since nobody cares?07:40
*** gildub has quit IRC07:42
*** salv-orl_ has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/project-config: Move py34-with-oslo-master jobs to xenial  https://review.openstack.org/37794707:44
*** links has joined #openstack-infra07:45
openstackgerritMerged openstack-infra/project-config: Open a repository for NFV filters.  https://review.openstack.org/37779407:45
openstackgerritMerged openstack-infra/project-config: gnocchi: make upgrade jobs voting  https://review.openstack.org/37828407:45
*** salv-orlando has quit IRC07:46
*** amitgandhinz has joined #openstack-infra07:46
*** gildub has joined #openstack-infra07:48
*** strigazi_AFK is now known as strigazi07:49
*** pilgrimstack has joined #openstack-infra07:50
*** sshnaidm has quit IRC07:51
*** ihrachys has joined #openstack-infra07:51
*** nadya has joined #openstack-infra07:51
openstackgerritMerged openstack-infra/project-config: Enable py35 voting on project zaqar  https://review.openstack.org/37814307:53
openstackgerritMerged openstack-infra/project-config: Add checks to openstack/cookbook-openstack-application-catalog  https://review.openstack.org/37767107:53
openstackgerritMerged openstack-infra/project-config: Create puppet-cloudkitty repository  https://review.openstack.org/36453507:55
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: WIP: Allow permissions to be set for teams in worklists and boards  https://review.openstack.org/37031207:58
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow permissions to be set for teams on stories  https://review.openstack.org/36891707:58
*** zzzeek has quit IRC08:00
*** yanyanhu_ has quit IRC08:01
*** hashar has joined #openstack-infra08:01
*** zzzeek has joined #openstack-infra08:01
*** sc68cal_ has joined #openstack-infra08:02
*** ijw has joined #openstack-infra08:03
*** Na3iL has joined #openstack-infra08:03
*** yaume has joined #openstack-infra08:03
*** sc68cal has quit IRC08:03
*** yanyanhu_ has joined #openstack-infra08:05
*** M-docaedo_vector has quit IRC08:05
*** [HeOS] is now known as HeOS08:09
*** watersoul has quit IRC08:11
*** ijw has quit IRC08:11
*** eharney has quit IRC08:16
openstackgerritMerged openstack-infra/project-config: Switch ironic-inspector to tempest discovery job  https://review.openstack.org/37762308:19
openstackgerritMerged openstack-infra/project-config: Convert tripleo-container jobs into multinode  https://review.openstack.org/37700008:20
*** esikachev has joined #openstack-infra08:21
*** links has quit IRC08:21
*** gildub has quit IRC08:21
*** amitgandhinz has quit IRC08:21
*** IzikP has quit IRC08:32
*** tqtran has joined #openstack-infra08:34
*** hferenc has joined #openstack-infra08:35
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add openstackci oscc cloud to Puppetmaster  https://review.openstack.org/37838408:36
*** akijak has left #openstack-infra08:37
*** links has joined #openstack-infra08:37
*** tqtran has quit IRC08:38
*** sdake has joined #openstack-infra08:41
*** derekh has joined #openstack-infra08:41
*** sdake has quit IRC08:42
*** sshnaidm has joined #openstack-infra08:42
*** jed56 has joined #openstack-infra08:45
*** chem has joined #openstack-infra08:45
*** mhickey has joined #openstack-infra08:46
*** ssancheztrujillo has quit IRC08:47
*** priteau has joined #openstack-infra08:48
*** openstackgerrit has quit IRC08:48
*** openstackgerrit has joined #openstack-infra08:49
*** claudiub|2 has quit IRC08:52
openstackgerritRob Cresswell proposed openstack-infra/irc-meetings: Update Horizon Meeting Chair to new PTL  https://review.openstack.org/37841708:54
*** _oanson is now known as oanson08:55
*** teran has quit IRC08:56
*** kaisers3 has quit IRC08:57
*** Julien-z_ has joined #openstack-infra08:57
*** Julien-z_ has quit IRC08:57
*** Julien-z_ has joined #openstack-infra08:58
openstackgerritRob Cresswell proposed openstack-infra/project-config: Make Horizon and D_O_A Django 1.10 tests voting  https://review.openstack.org/37842608:58
*** dtardivel has joined #openstack-infra08:58
*** kzaitsev_mb has joined #openstack-infra08:58
*** kaisers1 has joined #openstack-infra08:59
*** jamielennox is now known as jamielennox|away08:59
openstackgerritRodolfo Alonso Hernandez proposed openstack-infra/project-config: Add new project: devstack-plugin-libvirt-qemu  https://review.openstack.org/37792009:02
*** watersoul has joined #openstack-infra09:06
openstackgerritMerged openstack-infra/system-config: Add openstackci oscc cloud to Puppetmaster  https://review.openstack.org/37838409:07
*** claudiub|2 has joined #openstack-infra09:11
*** sambetts_ is now known as sambetts09:12
*** kaisers has quit IRC09:12
*** e0ne has joined #openstack-infra09:14
*** OferBarber has joined #openstack-infra09:15
*** pgadiya has quit IRC09:15
*** priteau has quit IRC09:15
Guest75624Where can I find results for Post jobs - e.g. the coverage jobs?09:16
*** priteau has joined #openstack-infra09:17
*** dmellado_ is now known as dmellado09:18
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackci launcher layouts  https://review.openstack.org/37845809:18
rcarrillocruzyolanda: mind reviewing ^ pls09:19
*** amitgandhinz has joined #openstack-infra09:19
*** rossella_ has quit IRC09:19
openstackgerritMerged openstack-infra/irc-meetings: Update Horizon Meeting Chair to new PTL  https://review.openstack.org/37841709:19
*** rossella_ has joined #openstack-infra09:20
*** yanyanhu_ has quit IRC09:24
*** shu-mutou is now known as shu-mutou-AWAY09:25
openstackgerritSam Betts proposed openstack-infra/irc-meetings: Add Networking Cisco IRC Meeting  https://review.openstack.org/37762209:26
*** pradiprwt has joined #openstack-infra09:27
yolandasure09:27
pradiprwtHi AJaeger09:28
pradiprwtPlease have a look to this bug review https://review.openstack.org/#/c/377639/09:29
rcarrillocruzthx09:32
*** pgadiya has joined #openstack-infra09:32
*** Julien-z_ has quit IRC09:35
*** tuannguyen has joined #openstack-infra09:36
pradiprwtHi rcarrillocruz, Please have a look to this bug review https://review.openstack.org/#/c/377639/09:36
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375009:37
openstackgerritMerged openstack-infra/system-config: Add chocolate openstackci launcher layouts  https://review.openstack.org/37845809:37
*** berendt has joined #openstack-infra09:39
*** electrofelix has joined #openstack-infra09:39
*** amoralej is now known as amoralej|out09:40
*** tuanla has joined #openstack-infra09:40
*** gongysh has quit IRC09:40
*** tuannguyen has quit IRC09:41
openstackgerritMerged openstack-infra/system-config: Add chocolate computes deployed and reachable from 21 to 30  https://review.openstack.org/37155109:44
*** denisra has joined #openstack-infra09:44
openstackgerritMerged openstack-infra/system-config: As initial step to deploy with HA, start using pacemaker module  https://review.openstack.org/33551109:45
*** electrofelix has quit IRC09:45
*** claudiub|2 has quit IRC09:45
*** electrofelix has joined #openstack-infra09:46
openstackgerritMerged openstack-infra/puppet-infracloud: Add the ability to configure bridge name  https://review.openstack.org/36869209:49
*** amitgandhinz has quit IRC09:51
*** denisra_ has joined #openstack-infra09:53
ralonsohHi. How can I add a PTL to a new group? According to http://docs.openstack.org/infra/manual/creators.html I need to ask you this09:54
*** matrohon has quit IRC09:56
*** denisra has quit IRC09:56
*** ralonsoh has left #openstack-infra09:59
*** ralonsoh has joined #openstack-infra09:59
*** M-docaedo_vector has joined #openstack-infra10:01
*** zhurong has quit IRC10:02
dimsAJaeger : we should ping harlowja and let him know. i'll try to remember to do that today10:07
AJaegerGuest75624: see http://docs.openstack.org/infra/manual/developers.html#automated-testing10:09
AJaegerralonsoh: tell us the review that merged and your email and I hope somebody from the team will read backscroll and add you.10:10
AJaegerthanks, dims10:11
ralonsohAJaeger: thank you. https://review.openstack.org/#/c/377794/. Email: rodolfo.alonso.hernandez@intel.com10:11
*** kaisers has joined #openstack-infra10:12
*** kaisers_ has joined #openstack-infra10:12
AJaegerinfra-root, please setup the groups for ralonsoh ^10:12
dimsAny git admins around? please nuke http://git.openstack.org/cgit/openstack/tacker-horizon/log/?h=stable/newton as i had a typo in my command line and ended up creating tag from an old SHA10:13
*** yamamoto has quit IRC10:13
*** kaisers__ has joined #openstack-infra10:14
*** kaisers_ has quit IRC10:14
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add mirror to chocolate openstackci cloud  https://review.openstack.org/37849910:15
*** kaisers has quit IRC10:16
*** prometheanfire has quit IRC10:17
openstackgerritSam Betts proposed openstack-infra/project-config: Add jobs to networking-cisco for testing version compat  https://review.openstack.org/37850210:17
*** esikachev has quit IRC10:18
*** prometheanfire has joined #openstack-infra10:19
*** sshnaidm is now known as sshnaidm|lnch10:21
*** ihrachys has quit IRC10:22
lennybwznoinsk, hi, faced same failures with devstack commit 69700227a9bdc65acd3aa8798e4eda7e8264dbb5. do you have any understandings of the issue?10:23
yolandaAJaeger, ralonsoh , let me check10:23
openstackgerritYuval Brik proposed openstack-infra/system-config: Rename Smaug to Karbor  https://review.openstack.org/35330910:24
wznoinsklennyb: hi, I guess depends what error youre seeing10:24
wznoinsklennyb: in my case the subnetpool IP addresses that are picked up sometimes clash with the subnet our external DNS is being configured in /etc/unbound/* inside the build VMs, subnetpool adds a route in the systems and connection to dns drops10:25
yolandadone10:25
lennybwznoinsk, :), I see that my servers is disconnected from the net during ./stack.sh ( replace ips ... ) and fails since it can't download required code10:25
wznoinskso probably similar story to mine lennyb, the route inside a VM added for subnetpool may be clashing with address you're using in your CI (public IPs of VMs?)10:26
AJaegeryolanda: do you have any idea why these puppet tests in  https://review.openstack.org/355588 fail? Is that a CI problem?10:27
yolandalooking10:28
lennybwznoinsk, I dont use vm, it's a physical server. I am checking how to configure properly/disable subnet pools10:28
yolandalint is a legit failure10:28
yolanda2016-09-27 07:30:35.471500 | manifests/bot.pp - WARNING: class included by relative name on line 810:28
yolandatrusty looks as a temporary failure, will need recheck10:29
rcarrillocruzpabelanger: would be good if we can get together and test https://review.openstack.org/#/c/373346/10:30
rcarrillocruzcos really, i lost like 20min getting auth errors when using openstackci user to do anything10:30
*** achevychalov has quit IRC10:31
wznoinsklennyb: https://github.com/openstack-dev/devstack/blob/69700227a9bdc65acd3aa8798e4eda7e8264dbb5/lib/neutron_plugins/services/l3#L85 here's the answer10:31
openstackgerritMerged openstack-infra/system-config: Add mirror to chocolate openstackci cloud  https://review.openstack.org/37849910:31
yolandaAJaeger, so no, trusty is also a legit failure, i got confused by the warnings about ec210:31
AJaegeryolanda: how do I replace "include gerritbot" ?10:31
rcarrillocruzturns out when you create the user with openstack client it won't add the user as member of the default project10:31
yolandareal failure is10:31
yolanda2016-09-27 07:22:14.446254 | Error: Invalid parameter nick on Class[Gerritbot] at /home/jenkins/workspace/gate-infra-puppet-apply-ubuntu-trusty/openstack-infra/system-config/modules/openstack_project/manifests/review.pp:318 on node ubuntu-trusty-osic-cloud1-4559008.openstack.org10:31
yolanda2016-09-27 07:22:14.446316 | Error: Invalid parameter nick on Class[Gerritbot] at /home/jenkins/workspace/gate-infra-puppet-apply-ubuntu-trusty/openstack-infra/system-config/modules/openstack_project/manifests/review.pp:318 on node ubuntu-trusty-osic-cloud1-4559008.openstack.org10:31
*** john-davidge has joined #openstack-infra10:31
yolandaAJaeger, do include ::gerritbot10:31
rcarrillocruzi.e. the 'Member' role has to be explicitly created and assigned10:31
rcarrillocruz /sadness10:31
AJaegerthanks, yolanda10:31
*** john-davidge has left #openstack-infra10:32
lennybwznoinsk, so setting SUBNETPOOL_PREFIX_V4=192.168.0.0 should solve the issue10:32
openstackgerritIsaac Beckman proposed openstack/diskimage-builder: Enable ssh password authentication  https://review.openstack.org/37657210:33
*** senk_ has quit IRC10:34
*** senk_ has joined #openstack-infra10:35
*** tuanla has quit IRC10:39
*** salv-orlando has joined #openstack-infra10:40
*** salv-orl_ has quit IRC10:40
*** _degorenko is now known as degorenko10:46
*** amitgandhinz has joined #openstack-infra10:47
*** links has quit IRC10:53
*** priteau has quit IRC10:53
wznoinsklennyb: yes, as long as you don't have anything inside your build environment that uses an IP from that subnet (like an external DNS server)10:57
wznoinskor proxy10:58
openstackgerritMarkos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration  https://review.openstack.org/37853211:00
*** dprince has joined #openstack-infra11:02
*** esikachev has joined #openstack-infra11:02
*** links has joined #openstack-infra11:02
openstackgerritBilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid  https://review.openstack.org/37853411:02
*** pgadiya has quit IRC11:06
*** sshnaidm|lnch is now known as sshnaidm11:08
*** senk_ has quit IRC11:09
*** sdague has joined #openstack-infra11:09
rcarrillocruzinfra-root , we haz chocolate mirror http://mirror.regionone.infracloud-chocolate.openstack.org/11:13
*** jkilpatr has joined #openstack-infra11:14
*** senk_ has joined #openstack-infra11:15
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Remove the mirror server resource from infracloud clouds  https://review.openstack.org/37854011:15
openstackgerritMarkos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration  https://review.openstack.org/37853211:15
*** rtheis has joined #openstack-infra11:16
*** kaisers__ has quit IRC11:17
*** senk_ has quit IRC11:19
*** pgadiya has joined #openstack-infra11:19
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add resources to chocolate openstackzuul cloud  https://review.openstack.org/37854211:20
*** ssancheztrujillo has joined #openstack-infra11:21
openstackgerritMarkos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration  https://review.openstack.org/37853211:21
*** amitgandhinz has quit IRC11:22
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackzuul oscc to Puppetmaster  https://review.openstack.org/37854511:23
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackci oscc to Puppetmaster all-clouds  https://review.openstack.org/37854611:25
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add resources to chocolate openstackzuul cloud  https://review.openstack.org/37854211:28
*** lucas-afk is now known as lucasagomes11:29
openstackgerritMerged openstack-infra/irc-meetings: Add Networking Cisco IRC Meeting  https://review.openstack.org/37762211:30
*** tuannguyen has joined #openstack-infra11:35
*** ccamacho is now known as ccamacho|lunch11:36
rcarrillocruzpabelanger: we'll also need https://review.openstack.org/#/c/378552/111:36
*** pilgrimstack has quit IRC11:36
rcarrillocruzos_keystone_role => create an openstack role11:36
*** pilgrimstack1 has joined #openstack-infra11:37
rcarrillocruzos_user_role => associates a role between a user and a project11:37
*** senk_ has joined #openstack-infra11:39
openstackgerritSam Betts proposed openstack-infra/project-config: Add jobs to networking-cisco for testing version compat  https://review.openstack.org/37850211:39
*** pilgrimstack has joined #openstack-infra11:40
*** pilgrimstack1 has quit IRC11:40
*** zhurong has joined #openstack-infra11:41
*** ldnunes has joined #openstack-infra11:42
*** pilgrimstack has quit IRC11:42
*** tuannguyen has quit IRC11:42
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add xenial tempest jobs  https://review.openstack.org/37736011:43
*** tuannguyen has joined #openstack-infra11:43
openstackgerritMerged openstack-infra/system-config: Add chocolate openstackzuul oscc to Puppetmaster  https://review.openstack.org/37854511:43
*** salv-orlando has quit IRC11:44
*** EricGonczer_ has joined #openstack-infra11:45
*** ianychoi_ has joined #openstack-infra11:46
openstackgerritMerged openstack-infra/system-config: Add chocolate openstackci oscc to Puppetmaster all-clouds  https://review.openstack.org/37854611:47
openstackgerritMerged openstack-infra/system-config: Add resources to chocolate openstackzuul cloud  https://review.openstack.org/37854211:47
*** tuannguyen has quit IRC11:48
*** thorst has joined #openstack-infra11:48
*** ianychoi has quit IRC11:49
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Remove the mirror server resource from infracloud clouds  https://review.openstack.org/37854011:49
*** derekh has quit IRC11:52
*** lezbar has quit IRC11:53
*** markvoelker has quit IRC11:53
*** lezbar has joined #openstack-infra11:53
*** berendt has quit IRC11:54
*** muawiakh_ has joined #openstack-infra11:55
*** muawiak__ has joined #openstack-infra11:55
*** muawiakhan has quit IRC11:56
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Add chocolate provider to Nodepool  https://review.openstack.org/37855911:56
*** Ravikiran_K has quit IRC11:57
*** danielitit has quit IRC11:58
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate oscc cloud to Nodepool  https://review.openstack.org/37856111:58
*** baoli has joined #openstack-infra11:58
*** muawiakh_ has quit IRC11:59
*** psilvad has joined #openstack-infra12:00
*** lucasagomes is now known as lucas-bbl12:01
*** sdake has joined #openstack-infra12:02
ttxGerrit is slow from here, anyone else noticing that ?12:04
*** rfolco_ has joined #openstack-infra12:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate params and resources to Nodepool node  https://review.openstack.org/37856412:05
*** psilvad has quit IRC12:06
vponomaryovttx: yes, 502 from time to time12:06
EmilienMttx: it's also slow here (Canada)12:08
openstackgerritMerged openstack-infra/system-config: Remove the mirror server resource from infracloud clouds  https://review.openstack.org/37854012:08
sdaguettx: yep12:09
*** edmondsw has joined #openstack-infra12:09
dulekttx: Slow for me too (EU).12:12
*** jpena is now known as jpena|lunch12:12
*** mtanino has joined #openstack-infra12:12
*** gordc has joined #openstack-infra12:12
ianychoi_Slow mee too12:12
ianychoi_s/mee/me12:12
*** ianychoi_ is now known as ianychoi12:12
*** amitgandhinz has joined #openstack-infra12:13
thervesdague, Thanks for you response on the infra ML12:14
thervesdague, 1) I asked for the image mirror. It was discussed during the meeting yesterday12:15
thervesdague, 2) I know that downloading stuff isn't great, but it didn't use to be that bad12:15
therve(3) I'm not subscribed to that list, sorry...)12:15
*** baoli has quit IRC12:16
*** psilvad has joined #openstack-infra12:16
sdaguetherve: yeh, I remember seeing similar issues with fedora mirrors a couple of years ago. I don't have much insight into how they are managed or monitored12:16
*** yamamoto has joined #openstack-infra12:16
*** mat128 is now known as mat128|afk12:17
thervesdague, As much as I'd like to blame them, it does sound like an openstack issue though :)12:17
mordredyah ... I think having an image mirror is going to be a good win12:17
sdaguetherve: could be, realize osic is 2/3 of our nodes12:18
therveUnless we got banned for too much download or something12:18
thervesdague, I was wondering, that's a good info, thanks12:18
mordredyah. I mean, that's why mirrors of things we use is so nice - it's also friendly to upstreams - we download things a LOT :)12:18
sdagueso only seeing it on osic is a pretty high statistical probability even if it's across the board issue12:18
ralonsohyolanda: Thank you!12:19
therveYeah, sorry we let that issue lingering for so long12:19
dmellado502 error and slow here too12:20
*** pilgrimstack has joined #openstack-infra12:21
*** vikrant has quit IRC12:22
openstackgerritBilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid  https://review.openstack.org/37853412:23
ttxinfra-core: gerrit might need a restart to free up memory12:24
rcarrillocruzhey12:24
rcarrillocruzyup, give me a sec12:24
*** amitgandhinz has quit IRC12:24
*** yamamoto has quit IRC12:25
*** rhallisey has joined #openstack-infra12:26
rcarrillocruzdone12:26
ianychoircarrillocruz, thanks! Seems to be much faster :)12:27
rcarrillocruzyeah, top said it was using max memory12:27
*** tuannguyen has joined #openstack-infra12:28
*** jamielennox|away is now known as jamielennox12:29
*** rlu has joined #openstack-infra12:30
*** narayrak has joined #openstack-infra12:30
*** bhavik1 has joined #openstack-infra12:31
*** bhavik1 has quit IRC12:31
*** tobasco has joined #openstack-infra12:32
*** salv-orlando has joined #openstack-infra12:33
*** bhavik1 has joined #openstack-infra12:33
*** amitgandhinz has joined #openstack-infra12:33
*** oanson has quit IRC12:34
*** Julien-zte has joined #openstack-infra12:37
*** mriedem has joined #openstack-infra12:37
*** e0ne has quit IRC12:40
openstackgerritMaciej Relewicz proposed openstack-infra/puppet-apps_site: Glare support for app-catalog  https://review.openstack.org/35902912:40
*** vponomaryov has quit IRC12:41
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375012:42
*** pzhurba has quit IRC12:42
*** aarefiev has quit IRC12:42
*** pzhurba has joined #openstack-infra12:43
*** e0ne has joined #openstack-infra12:43
*** vponomaryov has joined #openstack-infra12:43
*** rlandy has joined #openstack-infra12:44
openstackgerritAndrew Laski proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook  https://review.openstack.org/37711312:44
*** aarefiev has joined #openstack-infra12:44
*** yamamoto has joined #openstack-infra12:45
*** amitgandhinz has quit IRC12:45
*** yamamoto has quit IRC12:45
*** amitgandhinz has joined #openstack-infra12:45
*** amoralej|out is now known as amoralej12:46
*** rodrigods has quit IRC12:47
*** hashar is now known as hasharAway12:47
*** rodrigods has joined #openstack-infra12:47
*** trown|outtypewww is now known as trown12:48
kfox1111anyone have a good example of running a gate job that needs docker with centos7?12:48
kfox1111I'm running into the cloudinit+docker boot ordering issue I think.12:48
mordredkragniz: our gate nodes shouldn't have cloud-init on them12:49
*** yamamoto has joined #openstack-infra12:49
mordredgah. sorry kragniz12:49
kragnizhmm?12:49
mordredkfox1111: ^^12:49
kragnizah :)12:49
kfox1111oh... so maybe a different issue then.12:49
kfox1111http://logs.openstack.org/56/378056/7/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/b440a10/console.html12:49
kfox1111the sudo systemctl start docker runs but docker never starts.12:50
openstackgerritAndrew Laski proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook  https://review.openstack.org/37711312:50
mordredoh - boo. that job does not collect things like syslog and whatnot12:51
kfox1111just starting to write it. still figuring out what to do.12:52
kfox1111its my first.12:52
mordredkfox1111: so - there is a publisher macro called "devstack-logs" - ignore the name, it's not strictly for devstack12:53
mordredkfox1111: what it'll do is grab anything that's in the "logs" dir and publish it to logs.o.o with the console log12:53
mordredkfox1111: at the end of devstack-gate, we have some stuff that copies relevant log files into that logs dir12:54
mordredat the moment, that's not broken out for ease of reconsumption - although we just specced out doing that12:55
*** markvoelker has joined #openstack-infra12:55
* mordred is thinking through easiest way for you to add something similar for your job so that you can debug what's going on here ...12:55
*** EricGonc_ has joined #openstack-infra12:56
kfox1111yeah. that sounds good.12:56
mordredyah - I think it's devstack itself that copies syslog and other system logs into the logs dir12:57
*** Goneri has joined #openstack-infra12:57
*** david-lyle has joined #openstack-infra12:57
*** kaisers has joined #openstack-infra12:58
*** tuannguyen has quit IRC12:58
*** EricGonczer_ has quit IRC12:58
*** tuannguyen has joined #openstack-infra12:58
*** drawsmcgraw has quit IRC12:59
*** drawsmcgraw has joined #openstack-infra12:59
mordredkfox1111: ah - you do have the devstack-logs macro ... so all you need to do is add something to the end of your tox target that will copy logs into a dir named logs ... and make sure it does it whether things succeed or fail12:59
*** baoli has joined #openstack-infra12:59
mordred(you may want to make that only do so if a variable is set or something - not sure if devs run the tox target locally or not - if they do, it might be a really annoying addition :) )12:59
kfox1111`pwd`/logs? ~/logs? other?12:59
openstackgerritMarton Kiss proposed openstack-infra/groups: Add Flag module v3 commons refactor patch  https://review.openstack.org/37859613:00
*** tuannguyen has quit IRC13:00
*** tuannguyen has joined #openstack-infra13:00
*** baoli_ has joined #openstack-infra13:02
*** Julien-zte has quit IRC13:04
*** lucas-bbl is now known as lucasagomes13:05
*** baoli has quit IRC13:06
*** yamamoto has quit IRC13:07
kfox1111mordred: do you have an example of some code that copies in the logs?13:07
*** ccamacho|lunch is now known as ccamacho13:08
*** jamesdenton has joined #openstack-infra13:08
*** Julien-zte has joined #openstack-infra13:08
*** yamamoto has joined #openstack-infra13:08
*** mtanino has quit IRC13:09
*** sshnaidm is now known as sshnaidm|afk13:10
mordredkfox1111: effectively `pwd`/logs from the dir tox was executed from13:11
*** claudiub has joined #openstack-infra13:12
*** psachin has quit IRC13:12
mordredkfox1111: $WORKSPACE/logs actually - although it's the same thing13:13
*** woodster_ has joined #openstack-infra13:13
*** matt-borland has joined #openstack-infra13:14
kfox1111perfect. thanks. :)13:14
*** eantyshev has left #openstack-infra13:14
*** senk_ has quit IRC13:16
*** sshnaidm|afk has quit IRC13:16
*** senk_ has joined #openstack-infra13:16
*** berendt has joined #openstack-infra13:17
*** berendt has quit IRC13:17
*** kgiusti has joined #openstack-infra13:18
*** jkilpatr has quit IRC13:18
*** pzhurba has quit IRC13:18
*** jkilpatr has joined #openstack-infra13:18
*** rossella_ has quit IRC13:19
*** psachin has joined #openstack-infra13:19
*** rossella_ has joined #openstack-infra13:19
*** jaosorior has quit IRC13:20
pleia2fyi, gravestone unveiling and assorted related family things today, not around much13:20
*** jaosorior has joined #openstack-infra13:21
*** salv-orlando has quit IRC13:21
*** jpena|lunch is now known as jpena13:21
AJaegerclarkb: I'm happy now with vponomaryov's manila newton change - do you want to review it as well? https://review.openstack.org/#/c/37736013:23
*** Julien-zte has quit IRC13:23
*** Julien-zte has joined #openstack-infra13:24
*** mat128|afk is now known as mat12813:27
*** esberglu has joined #openstack-infra13:29
openstackgerritMonty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook  https://review.openstack.org/37173413:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner  https://review.openstack.org/36960413:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files  https://review.openstack.org/36960513:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async  https://review.openstack.org/37173313:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Rename callback_plugins dir to callback  https://review.openstack.org/37188713:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Use environment task parameter instead of parameters  https://review.openstack.org/36960213:30
openstackgerritMonty Taylor proposed openstack-infra/zuul: Rename zuul_runner to command  https://review.openstack.org/36960313:30
*** bhavik1 has quit IRC13:32
HeOSHello, infra team! If you have no objections, could you please merge this request: https://review.openstack.org/#/c/377639/ ?13:33
*** cardeois has joined #openstack-infra13:33
*** pgadiya has quit IRC13:34
*** cardeois has quit IRC13:36
*** esberglu has quit IRC13:36
*** esberglu has joined #openstack-infra13:36
*** tongli has joined #openstack-infra13:37
kfox1111mordred: ok, tried that:13:38
kfox1111https://review.openstack.org/#/c/378056/1013:38
kfox1111job failed with POST_FAILURE in 3m 25s (non-voting) now.13:38
kfox1111never seen that before.13:38
*** links has quit IRC13:38
kfox1111logs: http://logs.openstack.org/56/378056/10/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/7f56fb9/console.html13:38
*** tonytan4ever has joined #openstack-infra13:40
mordredkfox1111: wow... this is a fun one :)13:40
kfox1111:)13:40
mordredoh! you know ... this has nothing to do with the gate ...13:41
mordredbut I remember in my brainhole something wonky with docker and fedora/centos as it relates to selinux13:41
mordredwhich causes it to not be able to start13:41
kfox1111is selinux enabled on the boxes by default?13:41
*** cardeois has joined #openstack-infra13:41
*** esberglu has quit IRC13:42
pabelangergreghaynes: thanks!13:42
mordredI believe so .. pabelanger you know?13:42
kfox1111k. I'll add a disable in there...13:42
fungitherve: sheer speculation, but if it's ipv4 then maybe we're overloading their pat (since that environment only has ipv6 for direct global connectivity)13:42
fungidims: still need anything done with the tacker-horizon repo?13:43
*** roxanaghe has joined #openstack-infra13:43
*** eharney has joined #openstack-infra13:43
*** esberglu has joined #openstack-infra13:43
*** roxanaghe has quit IRC13:45
fungircarrillocruz: did you still want to work on the replacement cacti now that we have stats suggesting 4gb is probably sufficient? (after upping the connection table max and then looking at the system load and swap utilization, i' don't think a 2gb replacement will give us enough headroom)13:45
mordredkfox1111: https://github.com/docker/docker/issues/15498 other people having problems running docker on centos-7 in vms13:45
pabelangermordred: kfox1111: yes, selinux is enabled on centos-7 nodes13:45
mordredkfox1111: I have not read the whole issue yet - it's long - but it does have a bunch of debugging ideas13:45
kfox1111k. I'll ahve a look through.13:46
kfox1111I use it in vm's all the time on centos7 in our production systems.13:46
kfox1111which was why I was asking about cloud init.13:46
fungircarrillocruz: the gerrit jvm will use pretty much all of the system memory on review.o.o because it's sized to do so. performance will be determined by resource consumption _within_ the jvm, and for that you're pretty much stuck looking at the javamelody graphs for it13:46
mordredkfox1111: nod. well, I'm curious to know what the issue is :)13:47
kfox1111there's one issue where cloud-init happens before docker-storage-setup happens, os you cant do docker commands winthin cloud-init scripts.13:47
AJaegerfungi, could you put https://review.openstack.org/376573 on your review stack, please? That's a fix for the zuul-cloner change that broke all jobs on Monday...13:47
*** xyang1 has joined #openstack-infra13:47
kfox1111brb13:47
fungiAJaeger: lgtm, and great catch13:49
*** liusheng has quit IRC13:49
fungiAJaeger: as for changing the remaining jobs back to using that, we should probably wait a week. i'd rather avoid destabilizing things unnecessarily between now and release day just for the sake of consistency/cleanup13:50
AJaegerthanks, mordred and fungi for simultaneous +2s. I'll +A myself then ;)13:50
mordred++13:50
openstackgerritZhenguo Niu proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots  https://review.openstack.org/35558813:50
AJaegerfungi: YEs, definitely - and perhaps not everything at once ;)13:50
*** liusheng has joined #openstack-infra13:52
*** ddieterly has joined #openstack-infra13:52
openstackgerritSean Dague proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook  https://review.openstack.org/37711313:52
rcarrillocruzfungi: sure, i can work on that13:52
fungircarrillocruz: remember we'll want to copy over the rrd files from the old server to the  new one so we don't lose history13:53
rcarrillocruzis there anything wrong with the openstack_infra helper thing?13:53
fungiwhat helper thing?13:53
rcarrillocruzsec13:53
rcarrillocruzhttp://logs.openstack.org/64/378564/1/check/gate-openstackci-beaker-centos-7/cfd6901/console.html13:53
*** dprince has quit IRC13:53
rcarrillocruzgetting consistent failures on beaker tests13:53
fungioh, the gem13:53
rcarrillocruzfor some reason bundler rspec fails13:54
rcarrillocruzi've checeked the repo and it doesn't have changes lately13:54
fungiLoadError: cannot load such file -- serverspec13:54
rcarrillocruznot sure what's up13:54
*** dprince has joined #openstack-infra13:54
funginew bundler release?13:54
rcarrillocruzlet me see13:55
*** Guest92 has joined #openstack-infra13:55
kfox1111mordred: docker-storage-setup[2649]: INFO: Volume group backing root filesystem could not be determined13:55
kfox1111docker-storage-setup[2649]: INFO: Volume group backing root filesystem could not be determined13:55
kfox1111docker-storage-setup[2649]: ERROR: No valid volume group found. Exiting.13:55
kfox1111for some reason its not doing the loopback setup.13:55
kfox1111does the vm setup do any config of docker? cause that seems to be different then the distro defaults.13:56
rcarrillocruzlast release is from 15 days ago13:56
* rcarrillocruz scratches his head13:56
fungircarrillocruz: is it possible my change to un-pin bundler just merged?13:57
rcarrillocruzalso, fungi , pabelanger : we have chocolate mirror online13:57
mordredkfox1111: nope, no config of docker that I'm aware of13:57
kfox1111thats really strange....13:57
*** fguillot has joined #openstack-infra13:57
rcarrillocruzfungi: which change, you have it handy?13:58
fungircarrillocruz: no such luck, merged 12 days ago https://review.openstack.org/37106713:58
rcarrillocruzDate:   Fri Sep 16 02:11:24 2016 +000013:58
rcarrillocruzyeah13:59
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add osic-cloud1-(disk|s3500|3700) to grafana  https://review.openstack.org/37864213:59
*** mtanino has joined #openstack-infra13:59
*** hrybacki|afk is now known as hrybacki13:59
pabelangerrcarrillocruz: great14:00
mordredkfox1111: https://bugzilla.redhat.com/show_bug.cgi?id=1290283 see comment 614:00
openstackbugzilla.redhat.com bug 1290283 in docker "Docker Storage Setup enters failed state, is confused looking for non-existent VG "sda2"" [Unspecified,Closed: insufficient_data] - Assigned to vgoyal14:00
mordredkfox1111: (don't know if it's helpful or not)14:00
*** Rockyg has joined #openstack-infra14:00
*** andymaier has joined #openstack-infra14:01
*** isaacb has quit IRC14:01
fungircarrillocruz: the log says we're installing beaker-rspec 2.0.0, but https://rubygems.org/gems/beaker-rspec says 5.6.0 is current and 2.0.0 is from 201314:01
fungii wonder if someone flubbed a dependency in a new release of some other gem, or if latest bundler is still having problems resolving dependency versions correctly14:02
kfox1111its about someone who customized the docker storage setup. which I don't hink we have,14:02
kfox1111though its behaving like it has...14:02
fungircarrillocruz: any idea when that job started failing? just today?14:03
rcarrillocruzi noticed today yeah...14:03
rcarrillocruztwo different changes14:03
rcarrillocruzsame beaker job14:03
rcarrillocruzwith same error14:03
openstackgerritMerged openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/37742114:04
dimsfungi : yes please nuke the stable/newton from tacker-horizon14:04
*** mtanino has quit IRC14:04
fungircarrillocruz: i guess compare the bundle install list from a working and failing job to see what's changed (if anything)14:05
fungidims: i've deleted the stable/newton branch of openstack/tacker-horizon formerly at 865ef341a8380edc243c4c43d9c7943f6c26477914:06
AJaegerharlowja, http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/32d4c97/console.html is failing every time - looks like the tox.ini generated does not work. Could you check and fix it? Or should we remove it since nobody cares?14:07
rcarrillocruzwell14:08
*** sshnaidm|afk has joined #openstack-infra14:08
dimsthanks fungi14:08
rcarrillocruzso, from a failing job, the serverspec gem is not shown as installed14:08
rcarrillocruzwhereas a good one it does14:08
rcarrillocruzworking: http://logs.openstack.org/27/368727/4/check/gate-openstackci-beaker-centos-7/5596fa1/console.html.gz14:08
rcarrillocruznot working: http://logs.openstack.org/64/378564/1/check/gate-openstackci-beaker-centos-7/cfd6901/console.html14:08
fricklerrcarrillocruz: fungi: previous successful jobs had beaker-rspec 5.6.0. fwiw I was having issues with rubygems.org with other jobs lately14:08
cloudnullmornings14:09
rcarrillocruzfrickler: good, thanks14:09
*** xarses has quit IRC14:09
*** adrian_otto has joined #openstack-infra14:09
*** kaisers has quit IRC14:09
AJaegerproject-config cores, please review my changes for the track-upstream problems: https://review.openstack.org/378003 and https://review.openstack.org/378016 .14:10
*** gsilvis has quit IRC14:11
openstackgerritMerged openstack-infra/project-config: Fix zuul-release-git-prep-upper-constraints  https://review.openstack.org/37657314:13
*** rbrndt has joined #openstack-infra14:13
openstackgerritMonty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook  https://review.openstack.org/37173414:14
openstackgerritMonty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files  https://review.openstack.org/36960514:14
openstackgerritMonty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async  https://review.openstack.org/37173314:14
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries  https://review.openstack.org/37821514:16
anteayaAJaeger: i'm foggy on what this is fixing: https://review.openstack.org/#/c/378003/314:17
AJaegerclarkb: we check the upstream URL in tools/check_valid_gerrit_projects.py14:17
AJaegeranteaya: It's testing the URLs - we didn't do this so far.14:17
*** kaisers has joined #openstack-infra14:17
clarkbAJaeger: oh even better14:17
anteayacan you give me a scenario for why we want this?14:17
AJaegeranteaya: see 378016 for 5 repos that are broken. If we would have 378003 in, it wouldn't have happened.14:18
AJaegeranteaya: let me grab a change for you14:18
*** rlu has quit IRC14:18
AJaegeranteaya: see https://review.openstack.org/373202 - and https://review.openstack.org/377843, https://review.openstack.org/377117 - the last two wouldn't have happened if my change was in14:19
AJaegerclarkb: we can move the change from check_valid_gerrit_projects over to this. I wanted to do smaller steps ;)14:19
anteayasorry maybe it is because I haven't had any breakfast yet, I still don't follow14:19
cloudnullidk if anyone has a had a chance to look into this but in cloud1 we're seeing irregular error node counts http://grafana.openstack.org/dashboard/db/nodepool-osic?panelId=11&fullscreen And i'm not sure if its something on internal or elsewhere at this point.14:20
*** salv-orlando has joined #openstack-infra14:20
AJaegeranteaya: Currently, if you import a repo with upstream URL, we check the repo.14:20
anteayalast I understood normalize project config removed upstream url lines14:20
anteayacheck it for what?14:20
AJaegeranteaya: we check that it's properly setup14:20
*** nmagnezi has quit IRC14:20
AJaegerBUT we have with track-upstream the option to import a repo consistently into our setup.14:21
*** raildo has joined #openstack-infra14:21
AJaegerAnd those are not removed.14:21
AJaegerAnd those are not checked yet.14:21
anteayawas this always in place and I have a hole in my memory?14:21
AJaegeranteaya: look at the reviews. When there's track-upstream, then normalize does not remove them.14:21
AJaegeranteaya: yes - only user was openstack-infra/gerrit until two weeks or so ago.14:22
AJaegerNow we have 580+ users - and 7 of those were broken.14:22
anteayawhy the increase from gerrit to everyone14:22
AJaegerSo, I've written a check that zigo cannot break it...14:22
anteayaoh14:23
fungianteaya: it was only added for the deb package repos14:23
anteayaokay thanks14:23
anteayafungi: ah thank you14:23
AJaegeranteaya: that's the current debian package building setup.14:23
*** nadya has quit IRC14:23
*** hamzy has quit IRC14:23
fungitheir workflow basically has their packaging repos as forks of other git repositories, and they need to keep those forks up to date14:23
*** Julien-zte has quit IRC14:24
anteayaokay thank you14:24
anteayatrack-upstream only tracks repos on our git farm, yes?14:24
openstackgerritMerged openstack-infra/project-config: Check changed track-upstreams in gerrit/projects.yaml  https://review.openstack.org/37800314:24
*** hongbin has joined #openstack-infra14:24
mtreinishfungi: yeah, responses on our bug. \o/ Despite all our issues with mosquitto the maintainer is at least very active and helpful14:24
anteayanot github repos?14:24
fungiwhich is similar to how we track upstream gerrit's git repository in our gerrit fork14:24
*** Julien-zte has joined #openstack-infra14:24
anteayafungi: ah thank you14:24
fungianteaya: track-upstream can track any upstream git repo. for many of them we're also the upstream, for others we may not be14:25
*** xarses has joined #openstack-infra14:25
anteayaokay thank you14:25
*** rlu has joined #openstack-infra14:25
AJaegerfungi,clarkb: if you have ideas for checks that we should add for track-upstream repos - or for normal import ones - please tell. I can enhance the checks further...14:26
fungimtreinish: wow, that's a super-helpful triage14:26
anteayaAJaeger: so right now 'properly set up' means the url of the repo is accurate, yes?14:26
AJaegeranteaya: exactly. WE try a git clone on it. So, that's the most basic test.14:27
anteayaokay thank you14:27
*** yamahata has joined #openstack-infra14:27
zigoAJaeger: Thanks for writing this. It was very surprising to me to see that when track-upstream was set, there was no check. I was kind of relying on the checks. If I knew, I would have been a way more careful...14:27
*** gsilvis has joined #openstack-infra14:28
*** pfallenop has quit IRC14:28
openstackgerritMerged openstack-infra/project-config: Fix upstream tracking repos  https://review.openstack.org/37801614:28
*** lin_yang has joined #openstack-infra14:28
*** darvon has quit IRC14:28
zigoanteaya: FYI, our workflow is to "git merge -X thiers <FOO>" when upstream release a new release numbered FOO.14:28
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375014:29
zigoanteaya: Then git commit -a --amend to fix the packaging (like dependencies and version in debian/changelog).14:29
zigoanteaya: Then push the whole merge + packaging fix to Gerrit for review.14:29
mtreinishfungi: it does raise a question for the future of how we apply a fix once we get one. Presumably ubuntu will be slow to pull it into the package, if they ever do (older lts's never updated the mosquitto package)14:29
zigoanteaya: To be able to push a merge commit, we need the upstream tag and commits to be there already, otherwise gerrit rejects the "git review".14:30
zhenguoAJaeger: can you please help to approve this https://review.openstack.org/#/c/377268/? thank you14:30
zigoanteaya: If it was accepting it, we wouldn't need the track-upstream...14:30
zigoSo maybe there's room for improving Gerrit here.14:30
*** baoli_ has quit IRC14:31
EmilienMfungi, clarkb: good morning - FYI infra beaker jobs might break, see https://review.openstack.org/#/c/378648/ and https://github.com/puppetlabs/beaker-rspec/pull/8614:31
*** baoli has joined #openstack-infra14:31
AJaegerzhenguo: what is it about?14:31
anteayaEmilienM: thanks my skim of backscroll indicates we have been hitting beaker issues today14:32
anteayaEmilienM: thanks for sharing14:32
EmilienMcool np14:32
anteayazigo: thanks for the explanation14:32
zhenguoAJaeger: Add a project statusbot and meetbot :)14:32
*** muawiak__ has quit IRC14:33
AJaegerzhenguo: change for system-config? I don't review those...14:33
*** lennyb has quit IRC14:34
zhenguoAJaeger: yes, oh sorry I don't know about that14:34
openstackgerritEmilien Macchi proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases  https://review.openstack.org/37867014:35
EmilienManteaya: here's a quick fix ^14:35
anteayaEmilienM: thank you14:35
*** shardy has joined #openstack-infra14:36
fungiEmilienM: thanks, we had been trying to figure out what was going on there14:36
openstackgerritEmilien Macchi proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases  https://review.openstack.org/37867014:36
rcarrillocruzoh nice14:36
*** muawiakhan has joined #openstack-infra14:36
rcarrillocruzthanks EmilienM , i was getting that on my changes14:36
EmilienMhttp://logs.openstack.org/29/359029/34/check/gate-openstackci-beaker-centos-7/351d01d/console.html#_2016-09-28_14_01_58_72205514:37
fungiEmilienM: for some reason it was installing an ancient (2.0.0) beaker-rspec gem rather than the current version14:37
EmilienMI confirm we have the same problem in infra14:37
EmilienMfungi: yes!14:37
EmilienMso the PR is WIP in beaker-rspec14:38
EmilienMI would suggest you to merge my pin until it's solved upstream14:38
EmilienMif you want, i'll keep track on this PR and update both infra & puppet gemspecs when ready14:38
*** spzala has joined #openstack-infra14:38
*** baoli has quit IRC14:38
anteayaEmilienM: you're awesome14:38
anteayathank you14:38
fungithanks EmilienM, Hunner!14:38
EmilienMwell, trying to unblock CI14:38
clarkbcloudnull: pabelanger I can help take a look at those errors when properly awake14:38
anteayaEmilienM: and awesome14:39
*** caowei has quit IRC14:39
*** annegentle has joined #openstack-infra14:41
fungiclarkb: clarkb: pabelanger: any chance it's related to the reported network connectivity issues for jobs that are trying to download things from the internet at large?14:41
fungier, cloudnull ^14:41
fungiclarkb: cloudnull: pabelanger: http://lists.openstack.org/pipermail/openstack-infra/2016-September/004749.html14:42
clarkbmy guess is something related to our image shuffling yesterday14:43
clarkbI think your guess about NAT an ddownloads is likely14:43
fungiweird, when did github add gerrit-like inline commenting for code review?14:44
pabelangerclarkb: cloudnull: sorry, which errors?14:44
mordredfungi: a couple of weeks ago14:44
mordredfungi: they also added a "Review" primitive14:44
openstackgerritSarafraj Singh proposed openstack-infra/devstack-gate: Do not merge: Patch to test LM in nova  https://review.openstack.org/37801514:44
fungimordred: scary14:44
clarkbpabelanger: apparently new boot errors in osic14:44
mordredfungi: however, they did not seem to figure out that a reviewer might want to change their vote14:44
fungihah14:44
mordredfungi: OR that a new patchset might should clear out old votes14:44
pabelangerclarkb: Ya, just seen scrollback14:44
*** EricGonc_ has quit IRC14:44
pabelangerI can look now14:44
fungiclearly they hadn't ever used gerrit14:45
*** burgerk has joined #openstack-infra14:45
clarkbpabelanger: ok my hunch is its related to our image deduping somehow14:45
pabelangerOpenStackCloudException: Error in creating instance (Inner Exception: Can not find requested image (HTTP 400) (Request-ID: req-810b9f94-bf8e-4451-a515-ebce164c9113))14:45
pabelangeryup14:45
mordredfungi: and also clearly they didn't actually think through the workflow at all, since both of those are fairly obvious things, tbh14:45
fungimordred: sure. it seems like they implemented something based on a rough and incomplete description someone provided about how code review is typically done outside github14:46
*** Na3iL is now known as nzoueidi14:47
pabelangerclarkb: cloudnull: looks to be an issue with the ubuntu-xenial image not being found14:48
mordredfungi: well - you're generous there ... I'm guessing they implemented something without having considered that a world outside of github exists14:48
irtermitepabelanger: yea, I believe cloudnull brought that up late last night and asked if someone was going to upload it14:48
clarkbpabelanger: cloudnull can ptobably just queue a reupload and see if that solves it14:48
*** rlu has quit IRC14:49
clarkbbut maybe should consider undeduping if this sort of problem persists14:49
pabelangerI think I see the issue14:50
fungimordred: i have to assume that their _interest_ in adding code review at all was somehow informed by workflows outside github14:50
pabelangerI think osic-cloud1-s3700 uploaded a never image, which had a different ID14:50
*** adrian_otto has quit IRC14:50
pabelangerhttp://paste.openstack.org/show/583319/14:51
clarkbI did restart the builder while s3700 had a few images remaining14:51
clarkbin order to pick up centos7 and f24 fixes14:51
pabelangerso, maybe we delete the xenial images, and force an reupload14:51
*** yamamoto has quit IRC14:52
pabelangersee if nodepool does the right thing14:52
*** muawiakhan has quit IRC14:52
*** yamamoto has joined #openstack-infra14:52
clarkbpabelanger: ok14:53
clarkbI dont think we need to delete first though14:53
clarkbsince nodepool.should just use current image14:53
*** Swami has joined #openstack-infra14:53
*** kmartin has joined #openstack-infra14:54
anteayafungi: clarkb I'm going with logstash will fill up again before the end of today: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=422&rra_id=all14:55
*** ddieterly has quit IRC14:56
anteayathe good news is that the cacti graph doesn't seem to have gaps in it since fungi upped conntrack14:56
*** smarcet has joined #openstack-infra14:56
fungianteaya: yep, did you see my calculations yesterday? anyway we can go ahead and merge clarkb's change to disable debug logging if nobody has done so yet14:56
*** ddieterly has joined #openstack-infra14:56
anteayaI saw your calculations saying 34 hours from when you cleared it out14:56
anteayaI don't think we will make it to 34 hours14:57
fungikeep in mind that was based on an assumption that the activity volume i observed over a 5-minute period was representative14:57
anteayaoh yes, I'm not saying you are or were in accurate14:57
fungianyway, i've gone ahead and approved 37810414:57
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases  https://review.openstack.org/37867014:57
fungithat should stop it dead in its tracks14:57
*** hamzy has joined #openstack-infra14:57
anteayaI was attempting to say you were being generous14:58
anteayaobviously I missed14:58
anteayaand thank you14:58
fungioh, yes my "estimate" should have come with an implied +/- 24 hours ;)14:58
anteayaah14:58
*** Guest78091 is now known as redrobot14:59
anteayaperhaps it did in which case you are bang on14:59
fungiit was really an attempt to gauge the order of magnitude of log volume we were generating on that server14:59
anteayaindeed14:59
anteayareally big is a bit vauge14:59
anteayavague14:59
fungiso "fill up in a day" vs "in a week" or "month"14:59
*** yuval has quit IRC14:59
anteayayup15:00
*** adrian_otto has joined #openstack-infra15:00
*** armax has joined #openstack-infra15:01
*** zhurong has quit IRC15:01
anteayathe conntrack increase seems to have mostly worked, here is one with a gap that I believe occured since the change: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=30415:02
*** marst has joined #openstack-infra15:02
*** andreas_s has quit IRC15:02
*** lennyb has joined #openstack-infra15:03
openstackgerritMerged openstack-infra/puppet-log_processor: Reduce log client logging by default  https://review.openstack.org/37810415:03
*** r-mibu has quit IRC15:03
*** r-mibu has joined #openstack-infra15:03
*** rcernin has quit IRC15:04
anteayaeverything I have clicked on that isn't infra cloud seems to have been addressed by the conntrack increase15:04
anteayathis one is all over the place: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=30715:04
*** nzoueidi has quit IRC15:05
openstackgerritJp Maxwell proposed openstack-infra/system-config: Adding user 'maxwell' to OpenStack ID Production  https://review.openstack.org/37868615:05
*** jlanoux has quit IRC15:05
*** nzoueidi has joined #openstack-infra15:05
anteayathat seems to be par for the course for the infra cloud nodes, very spotty smtp reporting15:06
*** Swami_ has joined #openstack-infra15:06
*** tonytan_brb has joined #openstack-infra15:06
openstackgerritMonty Taylor proposed openstack-infra/shade: Make sure we're matching image status properly  https://review.openstack.org/37716415:06
openstackgerritMonty Taylor proposed openstack-infra/shade: Normalize images  https://review.openstack.org/37716315:06
*** caowei has joined #openstack-infra15:07
*** darvon has joined #openstack-infra15:07
*** pfallenop has joined #openstack-infra15:07
*** pfallenop has quit IRC15:07
*** pfallenop has joined #openstack-infra15:07
pabelangermordred: I think you were saying, shade will load an image into memory to calculate the md5sum? Is that something we could add to diskimage-builder? When we create the 1160.qcow2 image, also create 1160.qcow2.md5sum, and tell shade to use it?15:07
*** tphummel has joined #openstack-infra15:08
mordredpabelanger: sure. we'll need to calculate the sha256sum too15:09
pabelangerkk15:09
*** Swami has quit IRC15:09
mordredpabelanger: I think we may just want to have that be a nodepool thing though ... create_image takes an md5 and sha256 parameter which will cause it to not calculate those15:09
*** tonytan4ever has quit IRC15:09
pabelangerYa, that works15:09
*** GheRivero has joined #openstack-infra15:10
pabelangerclarkb: cloudnull: okay, ubuntu-xenial images sync'd. Checking logs now for more failures15:10
mordredpabelanger: so if we teach dib about making files and then teach nodepool to read those files and then pass the contents into the create_image call - I think that's the joy15:10
pabelangermordred: cool15:11
pabelangerclarkb: I do see some dupe images in osic-cloud1 (eg: nodepool-fedora-24-1148) http://paste.openstack.org/show/583328/ I wonder if we have a race going on15:12
*** kzaitsev_mb has quit IRC15:13
*** tdasilva has quit IRC15:14
anteayawhat does the number at the end mean? 1160, 1158, 1148 is that the incremented id of the creation of that image?15:14
clarkbpabelanger: ya I think its a race we dont serialize we will.have those15:14
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints  https://review.openstack.org/37804015:14
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test hostname map  https://review.openstack.org/37801715:14
clarkbpabelanger: I am also a little worried about the corresponding race on the delete side15:14
clarkbpabelanger: eg delete image 123 with uuid foo shared by image 15615:14
pabelangeranteaya: yes, that is the DIB id of said image15:14
anteayapabelanger: ah thanks15:15
clarkbnow 156 doesnt work if its upload hasnt finished for some reason. except we only delete the previous days so should be fine15:15
*** vinaypotluri has joined #openstack-infra15:15
clarkbI think in practice we willbe ok just need to watch for issues15:15
openstackgerritMerged openstack-infra/nodepool: Add missing month argument  https://review.openstack.org/36415115:16
*** thcipriani|afk is now known as thcipriani15:16
*** amotoki has quit IRC15:16
pabelangerclarkb: have we considered just adding a image-upload: True / False field for providers?15:17
pabelangeractually, not sure that would fix things either. Since provider would be pinned to a specific image15:17
pabelangernevermind15:17
*** nadya has joined #openstack-infra15:18
clarkbpabelanger: there are things we could do to only delete the image in glace if its uuid isnt in any other row in thr db15:19
clarkbI will ptobably work on that patch today15:19
*** zz_dimtruck is now known as dimtruck15:19
*** caowei has quit IRC15:20
clarkbI just dont know if it will be an actual issue in practice due to our keep two images policy15:20
dirkcan anyone help me with a 2nd pair of eyes on a jenkins job failure?15:20
dirkfor some reason I think I don't have python2.7 installed on the ubuntu-xenial image. can that be the case?15:20
anteayadirk: can you share the url for a log?15:21
dirkhttp://logs.openstack.org/50/378550/2/check/gate-requirements-tox-py27-with-upper-constraints-ubuntu-xenial/5346028/console.html15:21
dirkanteaya:15:21
dirkanteaya: the error message that caught my eye is15:21
*** lucasagomes is now known as lucas-hungry15:22
dirk/usr/local/jenkins/slave_scripts/run-tox.sh: .tox/py27-with-upper-constraints/bin/pip: /home/jenkins/workspace/gate-requirements-tox-py27-with-upper-constraints-ubu: bad interpreter: Permission denied15:22
*** yamahata has quit IRC15:22
anteayathanks I'm looking, I see the error, thank you15:24
pabelangerI suspect the shebang is too long (because of job name length) which is causing tox to fail15:25
pabelanger>128 path causes the failure15:26
pabelangeris it possible the job changed recently?15:26
*** tuannguyen has quit IRC15:26
*** apuimedo is now known as apuimedo|away15:27
*** rfolco_ has quit IRC15:27
jeblairclarkb: is it okay for me to stop nodepool, or are we uploading images you need?15:27
pabelangerI have some images I'd like to finish today15:28
*** mdrabe has joined #openstack-infra15:28
anteayadirk: yes it says it is installing python3, it doesn't say it is installing python2, I had thought python2 came with xenial15:29
dirkpabelanger: it is a recent problem, yes15:29
anteayadirk: python2 isn't installed by default on xenial: https://wiki.ubuntu.com/XenialXerus/ReleaseNotes/#Python_315:30
jeblairpabelanger: can you elaborate?15:30
dirkanteaya: great. so how do I tell $zuul to install it for me?15:31
*** baoli has joined #openstack-infra15:31
*** rfolco_ has joined #openstack-infra15:31
dirkI thought it magically detects that based on the job name15:31
pabelangerjeblair: we are doing builds with diskimage-builder 1.20.0 to include or ssh host key fixes, and new release of glean to fix rax-iad. So, would be nice to let them finish, however they could be pushed back another day or so if needed15:31
narayrakclarkb:rajinir: yesterdays CI issues15:32
narayrakWith the introduction of subnetpools, new variable SUBNETPOOL_PREFIX_V4 defaults to 10.0.0.0/8 which interfered our floating IP/network range and created routing issue15:32
narayrakhttps://github.com/openstack-dev/devstack/commit/69700227a9bdc65acd3aa8798e4eda7e8264dbb515:32
narayraknow have to handle SUBNETPOOL_PREFIX_V4, along with FIXED_RANGE15:32
narayrak+            local replace_range=${SUBNETPOOL_PREFIX_V4}15:32
narayrak+            if [[ -z "${SUBNETPOOL_V4_ID}" ]]; then15:32
narayrak+                replace_range=${FIXED_RANGE}15:32
narayrak+            fi15:32
narayrak+            sudo ip route replace $replace_range via $ROUTER_GW_IP15:32
*** rajinir has joined #openstack-infra15:32
anteayanarayrak: in future please use a paste service15:32
*** caowei has joined #openstack-infra15:32
jeblairpabelanger, fungi: okay, i may need someone else to take over the 'split-up-nodepool' work, or perhaps we should abandon it15:33
anteayadirk: well just to check pabelanger's theory do you have a log from a passing job?15:33
jeblairi'm in the worst timezone for it15:33
narayrakanteaya:Sure, thanks15:33
anteayanarayrak: thank you15:33
jeblairit was meant to be a quick temporary fix15:33
rajinirnarayrak:  This fixes the subnet issue with the CI builds https://review.openstack.org/#/c/378063/15:34
dirkanteaya: sure http://logs.openstack.org/77/339077/3/check/gate-requirements-tox-py27-with-upper-constraints/e620cdf/console.html.gz15:34
anteayathank you15:34
pabelangerjeblair: sure, I'm happy to wait another day for images to land the nodepool split15:34
dirkanteaya: its not passing but getting beyond that point15:34
*** smarcet1981 has joined #openstack-infra15:35
dirkanteaya: so yes, it got longer due to the ubuntu-xenial suffix15:35
*** jkilpatr has quit IRC15:35
jeblairpabelanger: well, i don't think you should have to wait.  it's possible to do both things at once, just not in my timezone.15:35
openstackgerritKyle Haley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template  https://review.openstack.org/37790815:35
*** ihrachys has joined #openstack-infra15:36
jeblairelectrofelix: can you look at https://review.openstack.org/378046 please?15:36
jeblairzxiiro: ^15:36
anteayadirk: so not passing on ubuntu xenial yet?15:36
cloudnullpabelanger: looks like the failure rate is dropping off.15:36
*** smarcet has quit IRC15:37
*** andymaier has quit IRC15:37
dirkanteaya: I think it was running on xenail before15:38
*** OferBarber_ has joined #openstack-infra15:38
dirkjust without the -ubuntu-xenial suffix15:38
clarkbjeblair: pabelanger we can always requeue images after any splitting15:38
dirkso this might mean that pabelanger is right15:38
*** ihrachys has quit IRC15:39
anteayadirk: I'm looking at http://logs.openstack.org/50/378550/2/check/gate-requirements-python27-ubuntu-xenial/def4397/console.html15:39
*** ihrachys has joined #openstack-infra15:39
anteayaa passing job from the same patch and I don't see in the log where python2.7 is installed15:40
pabelangerdirk: it should be easy to reproduce locally, just setup the same path locally, in your /tmp folder, then run tox.  You'll see the same bad interperter: Permission denied problem15:40
*** dimtruck is now known as zz_dimtruck15:40
*** OferBarber has quit IRC15:40
dirkanteaya: yep15:41
narayrakrajinir:thank you15:41
anteayaso yeah, right now I'm in agreement with pabelanger's theory15:41
anteayadirk: let me know how the local test goes15:41
kfox1111mordred: had a theory... maybe its as simple as docker not giving rights to the jenkins users to do docker ps? :/15:42
pabelangerclarkb: sure, I'm okay with that15:42
kfox1111testing now.15:42
mordredkfox1111: this is a good theory - you usually need to be in a special group to run docker commands15:43
kfox1111yeah.15:43
kfox1111didnt occur to me until just now.15:44
* mordred is sad he didn't think of it15:44
kfox1111me neither. andn I use  docker a million times a day. :)15:44
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Delete ping test environment in periodic jobs  https://review.openstack.org/34613415:44
dirkpabelanger: anteaya : bingo, yeap, it is the path length15:44
*** jlanoux has joined #openstack-infra15:44
anteayadirk: thank you15:45
anteayathat might be my first instance of job failing due to name too long15:45
openstackgerritKyle Haley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template  https://review.openstack.org/37790815:45
electrofelixjeblair: since JJB patches do get benefit from checking against a more complex set, we might need a replacement for just JJB, maybe just copy the existing into JJB for use as part of functional tests. Fully understand why removing the existing one makes sense for project-config15:45
*** Guest75624 is now known as BobBall15:45
dirkso we have to rename the job?15:45
openstackgerritJakub Libosvar proposed openstack-infra/project-config: Add scenarios from Neutron to multinode dvr full job  https://review.openstack.org/35534415:45
anteayadirk: yes15:46
anteayaapparently 63 characters is too long15:46
robcresswellCould really use a review on https://review.openstack.org/#/c/378406/ if possible; Horizons gate is totally locked up.15:46
anteayarobcresswell: what is going on that making the integration tests not voting is the way forward?15:47
*** tiswanso has joined #openstack-infra15:47
*** jkilpatr has joined #openstack-infra15:48
dirkanteaya: thanks15:48
anteayadirk: thank you15:48
robcresswellanteaya: Time, primarily. It's either hold everything until we can fix them, or a couple of people work on them and the other reviewers carry on as normal15:49
anteayabut what is the issue?15:49
anteayawhat is broken?15:49
robcresswellanteaya: We don't know yet. We've been getting intermittent failures across a broad range of tests15:49
anteayacan you share some logs?15:50
anteayaperhaps more eyes can assist15:50
anteayaI'm not comfortable making integration tests non voting15:50
*** mdrabe has quit IRC15:51
jeblairelectrofelix: makes sense.  should we proceed with landing the patch to disable while you work on the func testing?  or would you like me to make the compare-xml job work (by installing the 'afs' jjb module like we're doing in project-config)?15:51
*** mdrabe has joined #openstack-infra15:51
robcresswellHeh, Horizon PTL doesnt have control over its own tests now?15:51
robcresswellHere is one example http://logs.openstack.org/38/373738/3/gate/gate-horizon-dsvm-integration-deprecated-ubuntu-xenial/faf2992/15:52
anteayaI'm saying that integration tests affect many projects15:52
*** tuannguyen has joined #openstack-infra15:52
anteayaand that I'm not comfortable just being asked to toggle a switch15:52
openstackgerritDirk Mueller proposed openstack-infra/project-config: Rename py27-with-upper-constraints to py27-check-uc  https://review.openstack.org/37873215:52
*** tuannguyen has quit IRC15:53
electrofelixjeblair: yes, I don't see any reason to block, I'll look at what's needed for the func testing tomorrow15:53
jeblairelectrofelix: cool, thx15:53
dirkanteaya: pabelanger : your review appreciated :  https://review.openstack.org/37873215:53
openstackgerritMerged openstack-infra/project-config: Run Nova post_test_hook on placement job  https://review.openstack.org/37653715:53
*** tuannguyen has joined #openstack-infra15:53
robcresswellanteaya: I'm not sure I follow. It only affects the potential stability of Horizon or its plugins, nothing else.15:54
electrofelixjeblair: course the other alternative would be to teach JJB how to pick up extra modules in project-config automagically ;-)15:55
electrofelixbut I expect that would be just delaying the inevitable need to remove the compare job anyway15:55
anteayarobcresswell: so horizon integrated tests don't affect any other integrated projects?15:55
*** EricGonczer_ has joined #openstack-infra15:56
jeblairelectrofelix: yes.... and yes.  :)15:56
*** electrofelix has quit IRC15:56
robcresswellanteaya: No, they just run Horizon with whatever changes against devstack. Its not Tempest15:57
robcresswellJust end to end testing via selenium15:57
anteayadirk: why not just rename py27 with upper constraints in the tox file rather than adding an additional entry: https://review.openstack.org/#/c/378728/115:57
zxiiroelectrofelix: waynr  I'm at a conference this week so will miss the sprint. I think we still have many patches from the last sprint that needs review anyway. I've already reviewed most of them and can take a look again if necessary between flights.15:57
anteayarobcresswell: oh I beg your pardon then, my understanding of the use of integrated means 'all the projects tested together'15:58
*** tuannguyen has quit IRC15:58
dirkanteaya: hmm, I guess it depends on the merge order.. if the infra change gets merged first, I can just to a review that does the rename15:58
dirkI just added the depends-on as I don't know how likely it is that it would get merged otherwise when the tox.ini entry isn't there15:59
anteayadirk: ah okay15:59
anteayafair enough15:59
dirkin the past infra changes were not merged when the tox.ini did not exist properly15:59
*** tphummel has quit IRC15:59
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on Summit Event times getting local start/end date for summit events was changing the original value instead of clonnig it.  https://review.openstack.org/37874515:59
*** tonytan_brb is now known as tonytan4ever15:59
anteayadirk: how wise of us15:59
*** nadya has quit IRC16:00
dirkanteaya: yes, I don't argue against that :)16:00
openstackgerritMerged openstack-infra/openstackid-resources: Fix on Summit Event times getting local start/end date for summit events was changing the original value instead of clonnig it.  https://review.openstack.org/37874516:00
*** narayrak has quit IRC16:00
*** krtaylor has quit IRC16:01
*** slaweq_ has joined #openstack-infra16:01
*** narayrak has joined #openstack-infra16:01
anteayadirk: reviewed, I see a missed bit that needs to be removed16:01
*** berendt has joined #openstack-infra16:02
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/37743116:03
openstackgerritDirk Mueller proposed openstack-infra/project-config: Rename py27-with-upper-constraints to py27-check-uc  https://review.openstack.org/37873216:03
*** nadya has joined #openstack-infra16:03
*** mtanino has joined #openstack-infra16:04
openstackgerritMerged openstack-infra/nodepool: Use sequence znodes for image upload numbers  https://review.openstack.org/37104516:04
dirkanteaya: thx16:04
openstackgerritMerged openstack-infra/nodepool: Client change to support ZK image build requests  https://review.openstack.org/37176716:04
openstackgerritMerged openstack-infra/nodepool: Short-circuit builder processing on shutdown  https://review.openstack.org/37177016:04
*** caphrim007 has joined #openstack-infra16:05
anteayadirk: welcome16:05
openstackgerritMerged openstack-infra/nodepool: Handle ZooKeeper lost connections  https://review.openstack.org/37349916:05
openstackgerritMerged openstack-infra/nodepool: Handle config reload race  https://review.openstack.org/37441416:05
openstackgerritMerged openstack-infra/nodepool: Add UploadWorker skeleton to the builder  https://review.openstack.org/37706516:05
*** tuannguyen has joined #openstack-infra16:06
*** gyee has joined #openstack-infra16:07
*** ijw has joined #openstack-infra16:07
*** Julien-zte has quit IRC16:08
robcresswellthanks for the review anteaya16:08
anteayayou're welcome, thanks for explaining the situation to me, I appreciate it16:09
*** rossella_ has quit IRC16:09
openstackgerritMerged openstack-infra/project-config: Add JJB AFS module  https://review.openstack.org/37804516:09
*** eliqiao has quit IRC16:10
*** tdasilva has joined #openstack-infra16:11
*** thorst_ has joined #openstack-infra16:11
*** roxanaghe has joined #openstack-infra16:11
openstackgerritMerged openstack-infra/project-config: Stop running compare-xml jobs  https://review.openstack.org/37804616:12
openstackgerritMerged openstack-infra/project-config: Publish infra docs to AFS  https://review.openstack.org/37793016:12
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix token cache recursion  https://review.openstack.org/37875616:12
openstackgerritMerged openstack-infra/project-config: Enable layout-checks.py again  https://review.openstack.org/37827516:12
waynrzxiiro: thanks for the heads up16:12
openstackgerritMerged openstack-infra/openstackid-resources: Fix token cache recursion  https://review.openstack.org/37875616:13
*** thorst has quit IRC16:13
pabelangerdirk: anteaya: I'm going to defer to AJaeger on the upper-constraints job, I know he's been doing a lot of work on it16:14
dirkpabelanger: np16:14
*** njohnston has quit IRC16:15
openstackgerritBilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid  https://review.openstack.org/37853416:15
*** njohnston has joined #openstack-infra16:16
*** jpich has quit IRC16:16
anteayapabelanger: thanks for the pointer to the job name length16:16
*** ddieterly is now known as ddieterly[away]16:18
mordredclarkb: have you seen this: http://logs.openstack.org/49/377049/1/check/gate-dsvm-nodepool-src-shade/e7e180b/logs/screen-nodepool-builder.txt.gz#_2016-09-26_21_12_57_289 ?16:19
mordredclarkb: I'm seeing that on all of the nodepool/shade jobs on shade patches16:20
*** lucas-hungry is now known as lucasagomes16:20
mordredclarkb: oh - nevermind - wow, I see the real error16:20
*** mhickey has quit IRC16:20
*** jamielennox is now known as jamielennox|away16:21
*** strigazi is now known as strigazi_AFK16:21
mordredpabelanger, clarkb: http://logs.openstack.org/49/377049/1/check/gate-dsvm-nodepool-src-shade/e7e180b/logs/screen-nodepool-builder.txt.gz#_2016-09-26_21_12_56_70016:21
*** Guest46101 is now known as mgagne16:21
*** mgagne has quit IRC16:21
*** mgagne has joined #openstack-infra16:21
clarkbya it failed to connect for ubuntu image16:21
jeblair:(16:21
clarkbpossibly related to therves thing maybe16:21
mordredpossibly - it's also not using the pypi mirror in the dib build16:21
*** tuannguyen has quit IRC16:22
*** caowei has quit IRC16:22
*** njohnston has quit IRC16:22
mordredor maybe it would - it's failing downloading get-pip.py :)16:22
*** ddieterly[away] is now known as ddieterly16:22
*** tphummel has joined #openstack-infra16:22
*** njohnston has joined #openstack-infra16:23
*** yamamoto has quit IRC16:23
*** tdasilva has quit IRC16:23
*** njohnston has quit IRC16:25
greghaynesIf you all can find a way to mirror that it should be easy to make dib used the mirrored URI for it...16:25
*** flepied has quit IRC16:26
*** tuannguyen has joined #openstack-infra16:26
*** shardy has quit IRC16:26
*** jaosorior has quit IRC16:27
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website  https://review.openstack.org/37876916:28
*** ihrachys has quit IRC16:29
*** yaume has quit IRC16:29
*** thorst_ is now known as thorst16:30
*** njohnston has joined #openstack-infra16:30
*** nzoueidi has quit IRC16:30
*** njohnston has quit IRC16:32
*** njohnston has joined #openstack-infra16:32
*** njohnston has quit IRC16:33
*** narayrak has quit IRC16:33
*** Swami_ has quit IRC16:34
*** njohnston has joined #openstack-infra16:34
*** jlanoux has quit IRC16:34
*** baoli has quit IRC16:34
*** sshnaidm|afk is now known as sshnaidm16:34
*** baoli has joined #openstack-infra16:35
*** njohnston has quit IRC16:35
*** armax has quit IRC16:36
*** annegentle has quit IRC16:36
*** njohnston has joined #openstack-infra16:36
*** Ahharu has joined #openstack-infra16:37
*** njohnston has quit IRC16:37
Ahharuhi16:37
Ahharu2016-09-28 16:25:10.462691 | cp: cannot stat ‘/root/openrc’: No such file or directory16:37
Ahharuis this normal on beaker tests?16:37
Ahharuit just started bombing16:37
*** tuannguyen has quit IRC16:38
openstackgerritMonty Taylor proposed openstack-infra/shade: Add caching decorators to zones and record_sets  https://review.openstack.org/37806916:38
*** njohnston has joined #openstack-infra16:38
*** njohnston has quit IRC16:38
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website  https://review.openstack.org/37876916:39
anteayaAhharu: well https://github.com/puppetlabs/beaker-rspec/pull/86 is apparently a cause of new issues16:39
*** njohnston has joined #openstack-infra16:39
anteayaAhharu: what project are we discussing?16:39
Ahharuour project is `puppet-midonet`16:40
Ahharuhttps://review.openstack.org/#/q/puppet-midonet16:40
*** Apoorva has joined #openstack-infra16:40
openstackgerritMerged openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website  https://review.openstack.org/37876916:40
anteayahttps://review.openstack.org/#/c/378648/ merged two hours ago16:40
*** njohnston has quit IRC16:40
*** esikachev has quit IRC16:40
*** njohnston has joined #openstack-infra16:40
anteayawould that affect your project?16:40
*** njohnston has quit IRC16:42
*** njohnston has joined #openstack-infra16:42
*** tuannguyen has joined #openstack-infra16:42
*** amitgandhinz has quit IRC16:43
*** njohnston has quit IRC16:43
*** njohnston has joined #openstack-infra16:45
*** drifterza has quit IRC16:45
*** njohnston has quit IRC16:45
anteayaI'm going with you have enough to make you happy16:46
*** njohnston has joined #openstack-infra16:47
*** njohnston has quit IRC16:47
cloudnullpabelanger: clarkb: fungi: no errors in cloud1 for the last hour. seems like whatever was done has made things much happier.16:47
*** slaweq_ has quit IRC16:48
anteayacongratulations16:48
*** njohnston has joined #openstack-infra16:50
*** krtaylor has joined #openstack-infra16:50
*** njohnston has quit IRC16:50
mordred\o/16:50
*** njohnston has joined #openstack-infra16:51
anteayaAJaeger: I see https://review.openstack.org/#/c/376573 merged, thank you16:51
*** e0ne has quit IRC16:52
*** njohnston has quit IRC16:52
*** njohnston has joined #openstack-infra16:52
*** njohnston has quit IRC16:53
*** njohnston has joined #openstack-infra16:54
*** adrian_otto has quit IRC16:54
*** sandanar has joined #openstack-infra16:55
*** njohnston has quit IRC16:55
eharneydoes the requirements update bot not automatically bump the hacking version in test-reqs.txt?  i'd expect https://review.openstack.org/#/c/378793/ to contain the hacking<0.12 change16:55
*** njohnston has joined #openstack-infra16:55
*** muawiakhan has joined #openstack-infra16:55
openstackgerritMerged openstack-infra/system-config: Add chocolate oscc cloud to Nodepool  https://review.openstack.org/37856116:56
*** njohnston has quit IRC16:57
*** njohnston has joined #openstack-infra16:57
cloudnullno rush, but any chance on getting https://review.openstack.org/#/c/378642, https://review.openstack.org/#/c/378215, https://review.openstack.org/#/c/378100/ though?16:57
openstackgerritMerged openstack-infra/system-config: Add chocolate params and resources to Nodepool node  https://review.openstack.org/37856416:57
*** amitgandhinz has joined #openstack-infra16:58
*** njohnston has quit IRC16:58
anteayaeharney: any idea how long ago hacking changed? I'm trying to find the patch that changed it in requirements16:59
*** njohnston has joined #openstack-infra16:59
*** muawiakh_ has joined #openstack-infra16:59
*** muawiakhan has quit IRC16:59
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr: Updated from global requirements  https://review.openstack.org/37887016:59
clarkbhacking is special in requirements16:59
*** adrian_otto has joined #openstack-infra16:59
clarkbbecause its expected to fail when bumped and require intervention16:59
eharneyclarkb: ahh, i thought that might be the case.  would be a good thing to note in the bot's commit message maybe, when it happens16:59
anteayaah, thank you clarkb16:59
clarkbeharney: not sure what you mean the bot never makes that commit iirc17:00
*** njohnston has quit IRC17:00
eharneyclarkb: i meant a message like "hacking's version has changed in test-reqs.txt, but it has been skipped" in the "Updated from global reqs" commit17:01
*** njohnston has joined #openstack-infra17:01
clarkbgotcha, feel free to bring it up with the requirements team and or push a patch17:01
*** njohnston has quit IRC17:02
pabelangerwow17:02
clarkbeharney: not sure if they have other plans around that17:02
pabelangerwe just launched 1242 nodes in nodepool17:02
*** eliqiao has joined #openstack-infra17:03
pabelangerall from OpenStack proposal bot17:03
*** njohnston has joined #openstack-infra17:04
clarkbpabelanger: proposal bot launches nodes?17:04
pabelangerclarkb: sorry, the patchset from it did17:04
pabelangerhttps://review.openstack.org/#/q/topic:openstack/requirements17:05
*** njohnston has quit IRC17:05
*** smarcet has joined #openstack-infra17:05
*** muawiakh_ has quit IRC17:06
pabelangercool, quota issue in infracloud-vanilla17:06
*** njohnston has joined #openstack-infra17:06
pabelangerOpenStackCloudException: Error in creating instance (Inner Exception: Quota exceeded for ram: Requested 8192, but already used 794624 of 800000 ram (HTTP 403) (Request-ID: req-9c729032-060f-4199-a455-80e1d77972df))17:06
*** muawiakhan has joined #openstack-infra17:06
fungiyeah, when a requirements change lands, and the post job from that proposes requirements updates to other projects, each of those then spins up a number of check jobs17:06
fungiso tends to create a bit of a thundering herd17:06
*** muawiakhan has quit IRC17:07
*** njohnston has quit IRC17:07
openstackgerritMerged openstack-infra/project-config: Make Horizon integration tests non-voting  https://review.openstack.org/37840617:07
pabelangerYa17:07
pabelangerthat's the first time we launched all nodes in infracloud-vanilla17:07
*** psachin has quit IRC17:08
*** jpena is now known as jpena|off17:08
*** njohnston has joined #openstack-infra17:08
*** njohnston has quit IRC17:08
*** smarcet1981 has quit IRC17:08
openstackgerritMerged openstack-infra/project-config: Enable networking-ovn plugin on subnode config  https://review.openstack.org/37792117:09
anteayaAJaeger: you are correct, I should have had robcresswell remove the non voting jobs from the gate in https://review.openstack.org/#/c/378406/17:09
*** njohnston has joined #openstack-infra17:09
anteayarobcresswell: are you willing to offer a patch that cleans that up?17:10
*** flepied has joined #openstack-infra17:12
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Replace *_ssh jobs with *_ipmitool  https://review.openstack.org/37792817:14
*** ddieterly is now known as ddieterly[away]17:15
anteayaI got my acronym wrong for cacti data retrevial, I so can't acronym17:16
*** xarses has quit IRC17:19
pabelangerrcarrillocruz: I17:20
pabelangererr17:20
pabelangerrcarrillocruz: I've bumped the ram in infracloud-vanilla (openstackzuul) to 103219217:21
pabelangerthat lines up with 126 nodes at 819217:21
pabelangerrcarrillocruz: where did we land on setting quota in ansible?17:22
openstackgerritCory Benfield proposed openstack-infra/elastic-recheck: Prefer to invoke scripts directly without python.  https://review.openstack.org/37892517:22
*** yamamoto has joined #openstack-infra17:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Enable container job on multinode  https://review.openstack.org/28891517:25
*** berendt has quit IRC17:27
*** pvaneck has joined #openstack-infra17:27
*** sc68cal_ is now known as sc68cal17:27
*** Guest92 has quit IRC17:28
*** flepied has quit IRC17:29
*** sambetts is now known as sambetts|afk17:29
*** Swami has joined #openstack-infra17:31
*** yamamoto has quit IRC17:31
*** esberglu has quit IRC17:32
*** nadya has quit IRC17:32
*** esberglu has joined #openstack-infra17:32
*** tuannguyen has quit IRC17:33
*** apuimedo|away is now known as apuimedo17:34
*** xarses has joined #openstack-infra17:35
*** ralonsoh has quit IRC17:36
quade1AJaeger: Thanks for your help on this review. Do you know who I can reach out to to get Workflow+1? https://review.openstack.org/#/c/377908/17:36
*** nadya has joined #openstack-infra17:37
*** esberglu has quit IRC17:38
*** baoli has quit IRC17:39
*** baoli has joined #openstack-infra17:39
*** esberglu has joined #openstack-infra17:40
*** baoli has quit IRC17:40
*** baoli has joined #openstack-infra17:41
*** tuannguyen has joined #openstack-infra17:43
*** e0ne has joined #openstack-infra17:44
*** thorst has quit IRC17:45
*** thorst has joined #openstack-infra17:46
*** jimbaker` is now known as jimbaker17:48
*** quade1 has quit IRC17:48
*** amitgandhinz has quit IRC17:48
*** rbrndt has quit IRC17:48
*** quade has joined #openstack-infra17:49
*** amitgandhinz has joined #openstack-infra17:49
*** tpsilva has joined #openstack-infra17:50
*** fernnest has joined #openstack-infra17:50
*** fernnest_ has joined #openstack-infra17:50
*** lin_yang has quit IRC17:50
anteayafungi: 3 days ago someone uploaded a remove_device.php file to a forum thread for cacti that is 8 years old: http://forums.cacti.net/viewtopic.php?f=5&t=27713&start=1517:53
*** sshnaidm is now known as sshnaidm|afk17:53
anteayaseems like removing a device from cacti isn't in the source code, which I find odd17:53
*** adrian_otto has quit IRC17:54
funginormally they expect you to delete files off the filesystem17:54
*** adrian_otto has joined #openstack-infra17:54
anteayais that how we removed stale host entries that have been removed from our system config?17:54
*** adrian_otto has quit IRC17:55
fungii'm not sure, honestly. i haven't been doing it, so would need to explore17:56
anteayaokay yeah, that is what I got from your comment on friday17:56
anteayawas trying to be helpful17:56
anteayaif I'm wasting your time, I'll stop17:56
AJaegerquade: any project-config ;) core can help with 377909. Just ask here, those that review are on IRC. Somebody might read it...17:57
*** trown is now known as trown|lunch17:57
*** amitgandhinz has quit IRC17:58
quadeAJaeger: Thanks again! Any core interested in a Workflow? https://review.openstack.org/#/c/377908/17:58
*** amitgandhinz has joined #openstack-infra17:59
*** annegentle has joined #openstack-infra17:59
*** adrian_otto has joined #openstack-infra17:59
AJaegerpabelanger: could you look later at https://review.openstack.org/378062 , please? That'S a grafana change...17:59
*** mriedem has quit IRC17:59
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints  https://review.openstack.org/37804017:59
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test hostname map  https://review.openstack.org/37801717:59
clarkbAJaeger: reviewing the manilla change now18:00
*** lezbar has quit IRC18:01
clarkbthen cloudnull's changes18:01
pabelangerAJaeger: +218:01
*** lezbar has joined #openstack-infra18:02
*** sandanar_ has joined #openstack-infra18:02
AJaegerthanks, pabelanger and clarkb18:02
*** amitgandhinz has quit IRC18:02
*** amitgandhinz has joined #openstack-infra18:03
*** rwsu has quit IRC18:03
openstackgerritRob Cresswell proposed openstack-infra/project-config: Make Horizon and D_O_A Django 1.10 tests voting  https://review.openstack.org/37842618:04
*** esikachev has joined #openstack-infra18:04
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Add to my schedule optimization  https://review.openstack.org/37894418:04
openstackgerritMerged openstack-infra/openstackid-resources: Add to my schedule optimization  https://review.openstack.org/37894418:05
*** sandanar has quit IRC18:05
*** cardeois_ has joined #openstack-infra18:06
mtreinishfungi, nibalizer, clarkb, mordred, pleia2, rcarrillocruz: if you get a sec could you take a look at: https://review.openstack.org/#/q/topic:mosquitto-redux which should clean a few things up18:06
*** cardeoi__ has joined #openstack-infra18:06
mtreinishI also think part of the reason we're seeing the websocket crash a bunch on mosquitto is because the only code example we have in the docs uses websockets18:07
mtreinishone of the patches in that topic adds an ssl code example (without websockets) which maybe will help with that18:07
*** kzaitsev_mb has joined #openstack-infra18:08
*** cardeoi__ has quit IRC18:08
*** cardeois has quit IRC18:08
*** cardeois has joined #openstack-infra18:09
*** rbrndt has joined #openstack-infra18:10
*** cardeois_ has quit IRC18:10
clarkbpabelanger: cloudnull I am good with merging the rebalancing of osic max-servers if you are both ready18:11
clarkbwill review grafana changes first though18:12
*** ddieterly[away] is now known as ddieterly18:12
pabelangerclarkb: another 20mins and our last DIB for this day is finished, so if we could hold off on any nodepool restarts. But ready too18:12
pabelangerjust creating qcow2 now18:12
*** berendt has joined #openstack-infra18:13
clarkbpabelanger: the rebalancing of max-servers won't require any restarts18:13
clarkbits just shifting load between the logical osic cloud1 providers18:14
pabelangercoolio18:14
*** tqtran has joined #openstack-infra18:14
*** amoralej is now known as amoralej|off18:15
openstackgerritMerged openstack-infra/project-config: Show more graphs in a single row  https://review.openstack.org/37806218:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook  https://review.openstack.org/37173418:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner  https://review.openstack.org/36960418:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files  https://review.openstack.org/36960518:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async  https://review.openstack.org/37173318:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Rename callback_plugins dir to callback  https://review.openstack.org/37188718:16
openstackgerritMonty Taylor proposed openstack-infra/zuul: Rename zuul_runner to command  https://review.openstack.org/36960318:16
*** ijw has quit IRC18:16
*** tdasilva has joined #openstack-infra18:16
openstackgerritMerged openstack-infra/project-config: [Manila] Add xenial tempest jobs  https://review.openstack.org/37736018:17
*** jheroux has joined #openstack-infra18:18
*** tqtran has quit IRC18:18
*** Guest92 has joined #openstack-infra18:18
clarkbcloudnull: one minor thing noted on https://review.openstack.org/#/c/378215/218:19
*** Guest92_ has joined #openstack-infra18:20
clarkbcloudnull: you ready for me to approve the rebalancing now or do you want to get the grafana changes in first?18:20
irtermitehe's in a meeting clarkb18:20
irtermiteone sec18:20
*** degorenko is now known as _degorenko18:20
clarkbirtermite: ok no rush18:20
*** lezbar has quit IRC18:20
*** rossella_ has joined #openstack-infra18:20
*** lezbar has joined #openstack-infra18:21
irtermiteclarkb: go for rebalance; he doesn't need grafana quite yet18:23
*** Guest92 has quit IRC18:23
sridhar_raminfra-team: need a 2nd +2 for this change switching tacker python jobs voting https://review.openstack.org/#/c/370388/  ... anyone ?18:23
openstackgerritClark Boylan proposed openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries  https://review.openstack.org/37821518:24
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Add support for extra elastic-search graph filter  https://review.openstack.org/25558518:25
clarkbirtermite: I went ahead and just fixed the grafana thing since it was a one character change18:25
openstackgerritMerged openstack-infra/elastic-recheck: Make Elastic Recheck Watch more reusable  https://review.openstack.org/25151518:25
*** Jeffrey4l has quit IRC18:28
*** baoli has quit IRC18:28
openstackgerritMerged openstack-infra/project-config: Add osic-cloud1-(disk|s3500|3700) to grafana  https://review.openstack.org/37864218:28
irtermitethanks clarkb18:28
openstackgerritMerged openstack-infra/elastic-recheck: Make uncategorized_fails.py configurable  https://review.openstack.org/25007218:28
*** mriedem has joined #openstack-infra18:29
pabelangerclarkb: okay, ready to support. nodepool uploading DIBs now18:29
*** nadya has quit IRC18:30
mordredkfox1111: was that it?18:31
mrmartino/ hi18:31
mordredhey mrmartin18:32
*** krtaylor has quit IRC18:32
mrmartinhi mordred18:32
*** nadya has joined #openstack-infra18:32
mrmartinwho knows what is the current situation of stackalytics.o.o ?18:32
clarkbI think fungi and pabelanger probably have the best udnersatnd of the current state of stackalytics18:33
*** zz_dimtruck is now known as dimtruck18:33
*** ddieterly is now known as ddieterly[away]18:33
*** salv-orlando has quit IRC18:34
pabelangerYa, the work is done, its been running for a while. We just need to figure out the logistics on making the switch I think18:34
pabelangerI think we still need to do some tuning in apache2 too18:34
mrmartindo we have some tickets for the open issues in Storyboard?18:36
pabelangernothing outside of https://storyboard.openstack.org/#!/story/200027418:37
openstackgerritMerged openstack-infra/project-config: Update server distribution to HW specific entries  https://review.openstack.org/37810018:38
clarkbcloudnull: when you are out of meetings and have a moment I'd like to see if we can debug http://logs.openstack.org/19/373219/13/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/8f4250b/logs/devstack-subnodes-early.txt.gz basically ssh from one instance in osic cloud1 to another via the private ipv4 addresses seems to fail occasionally. We were seeing this on rackspace too18:38
mrmartinoh, ok I can allocate some time to finish this transition.18:38
irtermiteACK clarkb18:38
mrmartinso if we can collect what we need to do, I'm happy to help.18:38
*** pcrews has quit IRC18:39
fungipabelanger: mrmartin: i think the main issue we have is that a ton of stackalytics analysis is driven from overrides in "configuration" which it can only update by restarting. but it doesn't maintain a persistent data backend and has to rebuild all of its analysis from scratch to populate the in-memory database it uses, so restarts take something like half an hour (complete outage) to complete18:40
*** woodster_ has quit IRC18:40
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Enable more reasonable logging  https://review.openstack.org/36016418:40
fungiso we have a choice between frequent and lengthy outages for the service, or very infrequently applying updates to people's affiliation and address overrides18:41
*** ijw has joined #openstack-infra18:41
fungior rewriting a big chunk of stackalytics to use a persistent db and/or handle live configuration updates without restarting18:41
mrmartinfungi: oh, ok, I'll try to deploy it in a vm, and check how it works actually, and how can we improve this persistent data backend.18:41
mordredfungi: I thought it put all of its stuff into memcached18:41
openstackgerritMerged openstack/os-testr: Handle overlapping black regexes  https://review.openstack.org/37799918:41
openstackgerritMerged openstack/os-testr: Updated from global requirements  https://review.openstack.org/37887018:42
fungimordred: if so, then the cause of the lengthy outages to restart it may be for other reasons, but that was the story i had heard anyway18:42
mordredfungi: gotcha. I mean, I don't really actually know what I'm talking about :)18:42
clarkbfungi: anteaya the puppet change to remove debug logs on the jenkins log client is applied on logstash.o.o I will go ahead and restart the service to change its logging18:42
clarkbfungi: do you think I should also truncate the logs?18:43
fungiat least, i can tell you that if you restart stackalytics on stackalytics.o.o it's down for a long time, and the stackalytics.org maintainers have said they restart it manually on weekends when not as many people are looking at it18:43
rcarrillocruzpabelanger: https://github.com/ansible/ansible-modules-extras/pull/2709#issuecomment-24850135618:43
mordredahhh18:43
rcarrillocruzmeaning, it's probably going to be for 2.318:43
mrmartinyeah, ok18:43
rcarrillocruzso i tell you what18:43
*** yamahata has joined #openstack-infra18:43
mrmartincron can do magic18:43
rcarrillocruzi'll just hack around in the launcher parsing quota resources18:43
pabelangerfungi: we try to dump the data from memory if possible, but ya. It is an issue18:43
rcarrillocruzand run 'osc' commands in the background18:43
rcarrillocruzi'll let you know when i have something18:43
pabelangerrcarrillocruz: ah, okay18:44
rcarrillocruzfungi: in regards to the cacti replacement, should I put it on RAX_IAD or other region?18:44
fungipabelanger: oh, is it doing a pickle to disk or something?18:44
fungircarrillocruz: rax dfw, just like all our other control plane servers18:44
rcarrillocruzah yeah18:44
rcarrillocruzi keep on confusing regions18:44
rcarrillocruzthx18:44
fungime too, no worries18:44
pabelangerfungi: no, I wrote a crontab / logrotate to handle it: http://git.openstack.org/cgit/openstack-infra/puppet-stackalytics/tree/manifests/init.pp#n12718:45
fungiclarkb: we likely don't care about the logs from the past 24 hours, so no need to truncate them, just delete them?18:45
*** vsaienko has joined #openstack-infra18:45
rcarrillocruzthe way i did replacements in gozer was: create a cacti2 machine, add the puppet node, run puppet, do migration from cacti to cacti2, decommission cacti, point DNS of cacti.openstack.org to cacti2 machine, rename cacti2 to cacti on openstack . Is that how you usually do replacements?18:45
vsaienkodevstack-gate core: please help to review/merge https://review.openstack.org/#/c/33598118:45
clarkbfungi: ya I deleted them and service is restarted18:46
*** sdake has quit IRC18:46
*** ijw has quit IRC18:46
fungircarrillocruz: we just create a second one named cacti.openstack.org in nova and keep track of ip addresses until we update dns18:46
mordredI may not have realized you can rename servers in openstack though18:47
rcarrillocruzeugh, and doesn't the ansible wheel do weird things with two servers being named the same ?18:47
fungircarrillocruz: nope. ansible is working via uuid18:47
mordredor, it will for anything with duplicate names18:47
fungircarrillocruz: unless you want to do the additional work of making puppet handle maintaining/grouping cacti01 as well, and then we can leave the replacement named cacti01.openstack.org and create a cacti.openstack.org cname to it once it's running18:47
rcarrillocruzmordred: yeah, i even have some PR for an os_server_actions rename thing on my laptop18:47
rcarrillocruzi have to test it more thoroughly and push it, i hacked in it precisely having upgrades in mind18:48
*** krtaylor has joined #openstack-infra18:48
fungircarrillocruz: the problem with "renaming" servers after creation is that there are lots of places the original hostname ends up embedded on the filesystem18:48
rcarrillocruzyeah, an ansible playbook to reconcile new name would be good for that use case18:49
mordredyah18:49
rcarrillocruz/etc/hostname18:49
rcarrillocruz/etc/hosts18:49
*** rossella_ has quit IRC18:49
rcarrillocruzmail18:49
rcarrillocruzwhatever18:49
fungissh host keys too18:49
fungiand possibly lots of other stuff depending on what's initially installed18:50
rcarrillocruzso i guess that's part of the things to fix to end up with our goal of having uniquely named servers, yah?18:50
fungiin a lot of cases it's just embedded comments or other benign strings that don't influence operation18:50
pabelangerI thought doing cacti01.o.o and cacti02.o.o help solve some of those problems18:50
clarkbvsaienko: quick qestuon about that, set ips is set to false which means you won't have any l3 interface for that bridge on the test nodes. Which should mean you can't actually route to those IPs18:50
clarkbvsaienko: does that need to be set to True?18:50
*** rhallisey has quit IRC18:50
fungircarrillocruz: if you go the route with an ordinal suffix on the hostname (which i've done for a couple of our replacements now since we discussed it) you need to make sure to transition hiera from fqdn to a group, as well as making sure we get rid of any fqdn-isms in the puppet modules/classes we're using18:51
*** rhallisey has joined #openstack-infra18:51
clarkbvsaienko: or is l2 sufficient for you?18:51
vsaienkoclarkb: we tested this configuration it works just fine, we need only L2 https://review.openstack.org/#/c/368173/18:52
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Enable more reasonable logging  https://review.openstack.org/36016418:52
clarkbvsaienko: I only see failing jobs because devstack exercises could not be found?18:53
clarkboh thats multitest not multinode18:53
rcarrillocruzfungi: yeah, would be good to hack on that, cos i see there are streams of ansibling that machinery18:53
rcarrillocruzwhich can potentially be reused for future replacements18:53
vsaienkoclarkb: please have look at experimental multinode job: gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-multinode-nvSUCCESS in 49m 53s (non-voting)18:54
*** annegentle has quit IRC18:54
clarkbvsaienko: ya I am looking at it now18:54
clarkbvsaienko: this uses the ssh agent which ssh's to the host with the fake baremetal then boots using libvirt. Then from there how is ironic talking to it?18:55
vsaienkoclarkb: https://review.openstack.org/#/c/368647/3/devstack/lib/ironic@155118:56
*** lezbar has quit IRC18:56
*** baoli has joined #openstack-infra18:56
vsaienkoclarkb: we plug fake bare metal VMs to this network https://review.openstack.org/#/c/368647/3/devstack/lib/ironic@154518:57
*** lezbar has joined #openstack-infra18:57
clarkbvsaienko: right but what sort of communication is happening there? I am trying to figure out if that is required at all or if you can just use the neutron network. Because I am surprised if l2 without any ip addressing is suffiicent for you18:58
clarkbsince things like pxe require ip addresses iirc18:58
vsaienkoclarkb: I did near 10 recheck to make sure that conductor on primary node is able to talk to VMs located on subside18:58
clarkbvsaienko: yes I believe that it is working. I am wondering how18:59
vsaienkoclarkb: from the tunnel we need only L2, regarding to communication VM receives IP from neutron and than talks with conductor and API via L218:59
vsaienkoclarkb: I'm sorry via L318:59
clarkbvsaienko: what traffic goes over the ironic tunnel?18:59
vsaienkodhcp, pxe boot, http19:00
clarkbvsaienko: if neutron manages the l3 then why do we need a separate l2 network is I guess what this boils down to19:00
clarkbright thats not just l219:00
clarkbpxe and http require ip addresses19:00
clarkbI guess maybe the test nodes are arping on all interfaces?19:00
vsaienkoclarkb: I tried with native neutron tunnel, but it wasn't working, I think it is due to neutron ovs security flows19:00
*** hogepodge has quit IRC19:01
clarkbi would expect it to arp out the default route though if the ip addrs for the VMs aren't in a range it otherwise knows about which shouldn't get any ersponse19:01
vsaienkoclarkb: right, VM receives L3 info from Neutron, and that it communicates with ironic services via L319:01
*** spzala has quit IRC19:01
clarkbvsaienko: yes that part I understand. The part that is confusing me is how this l2 only bridge manages to do anything useful19:02
*** trown|lunch is now known as trown19:03
*** tuannguyen has quit IRC19:03
vsaienkoclarkb: it allows to pass L3 traffic between VM and host IP on another node (ironic api/conductor listening ) on that IP, but Neutron doesn't know about it19:04
clarkbvsaienko: but the nodes don't know about it either bceause you are not assigning an address to those bridges on the nodes19:04
clarkbvsaienko: the only things that will know about it are VMs on the bridge19:04
*** inc0 has joined #openstack-infra19:04
clarkbdoes ironic create a port itself for that?19:05
*** baoli has quit IRC19:05
clarkbs/port/interface/19:05
*** baoli has joined #openstack-infra19:06
*** tqtran has joined #openstack-infra19:06
vsaienkoclarkb: yes, I think this code you are asking for https://review.openstack.org/#/c/368644/3/devstack/lib/ironic@57919:06
pabelangerclarkb: mordred: this is from osic-cloud1 right now: http://paste.openstack.org/show/583365/19:06
*** dtardivel has quit IRC19:07
pabelangerguess we need to make nodepool check if image is currently queue / saving, and skip upload?19:07
*** berendt has quit IRC19:08
clarkbvsaienko: yes thats it. So you do have to assign an address you have just chosen to not do it in devstack-gate19:08
*** kzaitsev_mb has quit IRC19:08
rcarrillocruzpabelanger: in regards to launcher things, can you review https://review.openstack.org/#/c/378552/ ? for adding roles support, now with tests19:08
rcarrillocruzmordred: ^19:08
clarkbvsaienko: maybe you can leave a followup change to add a comment that describes that?19:08
kfox1111is there a way to check if a job is running or stuck or waiting to run?19:08
irtermiteIf you all see issues that need tracked or cloudnull or I are not around... feel free to drop an issue https://github.com/osic/osic-clouds19:08
vsaienkoclarkb: yes, I we wan't do it in devstack-gate because we already doing it in ironic19:08
kfox1111378056,19 has a 2hr 10min on it. and should onlyl take a few minutes to run.19:08
clarkbvsaienko: ya I think its fine, its just confusing fron devstack-gate's perspective19:09
vsaienkoclarkb: we don't want to assign in devstack-gatse19:09
kfox1111though I gues everythign else looks like its got similar runtimes on it...19:09
pabelangerrcarrillocruz: +2, the test helped19:09
clarkbkfox1111: click on the blue box and it will expand to give you job lists19:09
rcarrillocruzthx19:09
kfox1111mordred: not sure... jobs been stuck for a while. :/19:09
quadeAny cores willing to Workflow this bad boy? =D https://review.openstack.org/#/c/377908/19:10
rcarrillocruzi'll add tests for the user_roles change19:10
kfox1111clarkb: oh wow. didnt know yo ucould do that. :)19:10
kfox1111thanks. :)19:10
*** mhickey has joined #openstack-infra19:11
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: No need to make sure ZK lock path exists  https://review.openstack.org/37896119:11
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit  https://review.openstack.org/37511319:12
*** berendt has joined #openstack-infra19:12
*** yamahata has quit IRC19:13
vsaienkoclarkb: many thanks for your help, could you please help also to review/merge patches in the chain: https://review.openstack.org/#/c/352790 once you have a free time of course19:13
clarkbvsaienko: jlvillal has comments on that particular change19:14
fungiinfra-root: any objections to https://review.openstack.org/378686 ? (adds ssh access for jpmaxman to openstackid.org so he can double-up with smarcet on locally troubleshooting some elusive errors they're encountering under load)19:14
*** ddieterly[away] is now known as ddieterly19:14
*** cardeois has quit IRC19:15
vsaienkoclarkb: I will reupload last patch in the chain, there are two more patches https://review.openstack.org/#/c/364830 https://review.openstack.org/#/c/36861119:15
*** cardeois has joined #openstack-infra19:15
rcarrillocruzfungi: no objections, the more help the better19:15
clarkbvsaienko: ya reviewing them really quickly so you can update all in one go19:16
*** gyee has quit IRC19:16
mordredfungi: fine by me - I +2'd but did not +A so that you can gather as much feedback as you like19:16
*** pcrews has joined #openstack-infra19:17
vsaienkoclarkb: thank you, I will ping you when it done19:17
rcarrillocruzso folks, are we good to have chocolate on nodepool so we can get images building going ?19:17
rcarrillocruzhttps://review.openstack.org/#/c/378559/119:17
clarkbvsaienko: for the one that enables services you want the api to be running on the subnode too? I don't think it will be used there19:17
clarkbvsaienko: since we don't have a load balancer configured only the api services running on the primary node should show up in the catalog I think? maybe nova round robins them when talking to ironic though?19:18
vsaienkoclarkb: yeah, I'm going to remove api from subnode also19:18
*** shu-mutou-AWAY has quit IRC19:20
rcarrillocruzclarkb , pabelanger , fungi  ^19:20
clarkbvsaienko: looks like you have overloaded what multihost means too?19:21
clarkbvsaienko: multihost should specifically mean nova's multihost setting which is not useable with neutron and ironic aiui19:21
pabelangerrcarrillocruz: sure, but I'd like to hold off on restarting nodepool-builder until later tonight19:22
*** salv-orlando has joined #openstack-infra19:22
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data  https://review.openstack.org/37896919:22
rcarrillocruzk, since you'll be around after i go to bed, i leave that to you approving then?19:22
*** sandanar_ has quit IRC19:22
pabelangerrcarrillocruz: Ya, +3 now, I'll restart the builder later tonight19:23
rcarrillocruz++, thanks19:23
*** vsaienko has quit IRC19:24
*** vsaienko has joined #openstack-infra19:25
mtreinishclarkb: https://newsroom.t-mobile.com/news-and-blogs/t-mobiles-endless-summer.htm 2 weeks too late. That would have been nice to have in germany19:25
alexm2AJaeger: could you add workflow to https://review.openstack.org/#/c/377908/ please?19:25
AJaegeralexm2: no.19:26
*** tqtran has quit IRC19:26
AJaegeralexm2: two cores need to +2.19:26
*** kzaitsev_mb has joined #openstack-infra19:26
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data  https://review.openstack.org/37896919:26
clarkbmtreinish: will come in handy in barcelona though19:26
*** eeiden has quit IRC19:26
*** skipp has quit IRC19:26
*** stew925 has quit IRC19:26
*** wcriswell has quit IRC19:26
alexm2AJaeger, got it. Who can I ask? :)19:26
*** vsaienko has quit IRC19:28
*** Guest92_ has quit IRC19:28
*** vsaienko has joined #openstack-infra19:29
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data  https://review.openstack.org/37896919:30
*** tqtran has joined #openstack-infra19:31
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data  https://review.openstack.org/37896919:32
*** Guest92 has joined #openstack-infra19:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Create Member role on infracloud clouds  https://review.openstack.org/37897619:33
*** sdake has joined #openstack-infra19:34
openstackgerritEmilien Macchi proposed openstack-infra/system-config: Mirror centos virt mirror to AFS  https://review.openstack.org/37897819:35
EmilienMdmsimard: ^19:35
* clarkb wanders off to find lunch19:35
*** lezbar has quit IRC19:36
openstackgerritMerged openstack-infra/project-config: Add chocolate provider to Nodepool  https://review.openstack.org/37855919:36
*** lezbar has joined #openstack-infra19:36
fungialexm2: i'm curious, what is JIRA:NCP-2067 supposed to mean in the commit message of https://review.openstack.org/377908 ?19:39
alexm2fungi: it's our internal ticketing system id19:40
fungiit seems a bit strange to make vague references to what i can only assume is an unreachable, proprietary bug tracker in an upstream repo of a free software project19:40
fungialexm2: do you mind if i edit the commit message to remove that before i approve the change?19:41
alexm2fungi: go ahead19:41
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template  https://review.openstack.org/37790819:42
*** eeiden has joined #openstack-infra19:43
*** salv-orl_ has joined #openstack-infra19:43
openstackgerritMerged openstack-infra/tripleo-ci: tripleo.sh: log overcloud deploy command arguments before deploy  https://review.openstack.org/37063219:43
alexm2fungi: thanks!19:44
fungiyou're welcome19:44
*** nherciu has quit IRC19:44
*** oanson has joined #openstack-infra19:45
*** skipp has joined #openstack-infra19:45
*** stew925 has joined #openstack-infra19:46
*** camunoz_ has joined #openstack-infra19:46
*** salv-orlando has quit IRC19:46
*** wcriswell has joined #openstack-infra19:46
sdagueum, what's up with the graphite graphs for nodepool?19:48
pabelangermordred: clarkb: jeblair: now that we have runtime-ssh-host-keys in diskimage-builder, I'd like to land: https://review.openstack.org/#/c/374371/ which removes that functionality from glean19:48
sdagueit's like 1000 nodes available but jobs waiting an hour to get a test node?19:48
pabelangersdague: we suspect nodepool cannot keep up with the demand now. jeblair has some patches to break nodepool into multiple daemons now19:49
*** dimtruck is now known as zz_dimtruck19:49
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add imageUploadLock() API  https://review.openstack.org/37898319:49
*** vsaienko has quit IRC19:50
*** vsaienko has joined #openstack-infra19:51
*** camunoz_ has quit IRC19:51
*** rwsu has joined #openstack-infra19:53
fungijeblair: pabelanger: all the split-daemon patches so far are merged at this point, right? i'm not finding any more of them open for review at least19:54
pabelangerfungi: clarkb: apparently ssh-keygen -A requires some keyboard inputs: http://paste.openstack.org/show/583376/19:56
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic  https://review.openstack.org/36483019:56
pabelangerfungi: clarkb: a quick googles suggests doing $ yes n | ssh-keygen -A19:56
pabelangerto skip overriding of keys19:56
pabelangeroverwriting*19:56
pabelangerfungi: I think all patches are merged19:57
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Update ENABLED_SERVICE on subnode with ironic  https://review.openstack.org/36861119:57
pabelangerfungi: clarkb: I never seen the prompt in my testing, because I was also running: https://review.openstack.org/#/c/374371/19:58
*** vsaienko has quit IRC19:59
fungipabelanger: oh! got it. in your testing there weren't existing host keys because dib was deleting them and glean was not readding them before your initscript ran ssh-keygen20:01
*** spzala has joined #openstack-infra20:01
greghaynesIt'd be really neat to have some kind of functional testing for this20:02
greghaynesI dont think it'd be too hard to do either20:02
fungipabelanger: but without 374371 in production glean is (at least sometimes) creating host keys before the initscript runs ssh-keygen, and then you get prompted to overwrite them20:02
pabelangerfungi: correct, I'm testing yes n | ssh-keygen -A now, but think that is the way to disable the prompt20:03
*** kzaitsev_mb has quit IRC20:03
fungipabelanger: and i agree, there doesn't seem to be an option to ssh-keygen to force overwriting :/20:03
greghaynesI dont think you want force overwriting?20:03
*** berendt has quit IRC20:04
fungiwell, optionally we can make that initscript only create host keys when they don't already exist, correct20:04
*** lucasagomes is now known as lucas-afk20:04
greghaynesright, if you force overwrite I think we break users who want to reboot and not change hostkeys20:04
fungiso the initscript can just no-op and succeed if the keys are already there, and that satisfies the dependency the sshd initscript has20:04
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Handle ssh-keygen prompts for existing keys  https://review.openstack.org/37898520:05
pabelangerneed to build an image to test that20:05
fungigreghaynes: i agree. in our case (where the machine isn't reused and we don't support subsequent rebooting) it's mostly irrelevant, but in a general sense the initscript should no-op when the keys already exist20:05
greghaynesactually, wait a sec - http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/runtime-ssh-host-keys/init-scripts/systemd/ssh-keygen.service20:05
greghaynesThose ConditionPathExists should prevent this from happening20:06
pabelangergreghaynes: right, so far I am only seeing this in ubuntu-trusty20:06
pabelangerwhich is upstart20:06
greghaynesah20:06
*** spzala has quit IRC20:07
*** berendt has joined #openstack-infra20:07
*** salv-orl_ has quit IRC20:07
*** salv-orlando has joined #openstack-infra20:08
fungino way to make upstart do the same?20:08
pabelangersadly no20:08
*** ddieterly is now known as ddieterly[away]20:09
pabelangerwe'd have to add our logic as a pre-script I think20:09
*** inc0 has quit IRC20:09
*** flepied has joined #openstack-infra20:09
pabelangerbuild started20:10
openstackgerritMerged openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/37878820:11
pabelangerclarkb: I think the scenario you mentioned this morning about deleting an DIB just happened20:15
pabelangerthat is why we are seeing a lot of failures in osic-cloud1-s350020:15
pabelangerclarkb: actually, the image is still there... Hmm20:16
*** ddieterly[away] is now known as ddieterly20:17
pabelangerokay, osic-cloud1 uploaded a newer one20:18
pabelangerI've deleted it, since that region is going offline20:18
*** oanson has quit IRC20:20
*** roxanaghe has quit IRC20:22
*** tonytan4ever has quit IRC20:23
*** mhickey has quit IRC20:23
*** annegentle has joined #openstack-infra20:25
*** Goneri has quit IRC20:27
*** baoli has quit IRC20:29
*** trukise has joined #openstack-infra20:29
pabelangerfungi: yes n | ssh-keygen -A didn't work as expected20:30
pabelangerfungi: I guess we could not use ssh-keygen -A and manually generate any files that don't exists20:31
openstackgerritMerged openstack-infra/elastic-recheck: Prefer to invoke scripts directly without python.  https://review.openstack.org/37892520:31
fungipabelanger: have you ruled out being able to test for file existence in the upstart config? do we just need it to wrap a "very small shell script"?20:32
mordredpabelanger, greghaynes, fungi, clarkb: btw - I spoke with Jan van Eldik from CERN at OpenStack Nordic ... he's potentially interested in using glean there20:32
greghaynesooo20:33
pabelangerfungi: that is what I am testing now,20:33
fungipabelanger: oh, i see above you already said that, something about it having to be a pre-script20:33
*** Apoorva_ has joined #openstack-infra20:33
mordredin chatting with him, the split between the things in the simple-init element and the things in the glean repo struck me and made me wonder if we're making it harder for people to use it - but I betcha we can just get feedback from Jan on that20:33
fungimordred: woah! for science!!!20:33
mordredyah20:33
greghaynesmordred: yea, I can definitely see that20:34
*** ddieterly is now known as ddieterly[away]20:34
*** ddieterly[away] is now known as ddieterly20:34
*** kzaitsev_mb has joined #openstack-infra20:34
greghaynesIMO dib should just be some installer glue for the thing so the more we can make glean responsible for there the better20:34
greghaynesrandom thought - maybe the new ssh-key element can grab the init scripts from glean20:35
*** Apoorva has quit IRC20:36
mordredhrm - maybe so20:36
fungiwell, part of teh challenge here is that dib is clearing out those keys, sshd on most platforms won't start without them present, so we need the sshd initscript-like-things to be patched to depend on the key creation initscript-like-things20:37
*** adrian_otto has quit IRC20:37
clarkbsshd will start it will just derp connections and not be useful20:38
fungiso if dib is installing openssh-server or whatever, it should probably be responsible for patching its initscripts, which only makes sense if it also ships the initscripts it's patching them to depend on20:38
greghayneshrmmm20:38
mordredAFS isn't the solutoin to this is it?20:39
clarkbI thought we didnt need to patch sshds scripts20:39
clarkbwe just add init things that ho before sshd20:39
fungimordred: only if coupled with nntp20:39
clarkb*go before20:39
fungiclarkb: can you do that with lsb initscript headers? or are we really only needing to be able to declare it with systemd and upstart, not sysvinit?20:40
clarkbnot sure about sysv but none of the distros we support sysv so not sure if it matters20:40
fungiokay, so at least with upstart and systemd we should be able to declare a "before" relationship in the keygen init config20:41
clarkbyes20:41
fungiin which case ignore what i said about patching sshd initscripts20:41
fungioldthink20:41
pabelangerfungi: we can likely just do something like: http://paste.openstack.org/show/583381/ for upstart, rather depending on -A20:41
fungii suppose the glean package could just ship them in tat case, yes20:42
*** berendt has quit IRC20:42
jeblairpabelanger: what's the status of the image uploads you were waiting on?20:42
pabelangerjeblair: finished20:43
fungipabelanger: that's probably fine, yes20:43
jeblairso, okay to restart nodepool soon?20:43
*** ddieterly is now known as ddieterly[away]20:43
pabelangerjeblair: okay on this side20:43
fungii'm fine with a nodepoold restart any time20:43
jeblairok, i'll do that in around 20 mins20:43
*** rfolco_ has quit IRC20:44
*** tpsilva has quit IRC20:44
*** jamielennox|away is now known as jamielennox20:45
*** jkilpatr has quit IRC20:45
cloudnullirtermite: clarkb: sorry been in meetings for a good part of today. I'm looking into the "Failed to connect to the host via ssh" issue now.20:46
mordredcloudnull: I'd like to suggest that meetings are not a great use of your time :)20:47
mordredcloudnull: if it would be helpful for me to express that to other people, I'd be happy to20:47
cloudnull++ I completely agree.20:47
* mordred is good at loudly expressing things at people20:47
cloudnulli have a flag on my desk that says "I survived another meeting that should've been an email".20:47
cloudnullsadly it does not get the point across.20:48
mordredcloudnull: maybe you should schedule a meeting to discuss that point20:48
*** tuannguyen has joined #openstack-infra20:49
*** spzala has joined #openstack-infra20:49
cloudnullthat is a good idea.20:49
*** burgerk has quit IRC20:49
*** esikachev has quit IRC20:50
fungiback when i was forced to maintain a schedule in a group calendaring system, i would preemptively book most of my days and leave a few slots open where people could invite me to meetings20:50
cloudnullwe might need to schedule a meeting to talk about the meeting to discuss the points about why I dont need to be in meetings all day20:50
fungithe majority of my schedule was "i'm getting work done, don't interrupt me"20:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add cacti2 replacement node to Puppet  https://review.openstack.org/37899620:52
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for upstart  https://review.openstack.org/37898520:52
*** spzala has quit IRC20:53
*** nadya has quit IRC20:53
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/layout/puppet: run scenario003 for puppet-designate  https://review.openstack.org/37899720:54
EmilienMdmsimard: ^20:54
johnsomHi infra folks.  Diskimage-builder is grabbing pip from https://bootstrap.pypa.io/get-pip.py in this element: https://github.com/openstack/diskimage-builder/blob/master/elements/pip-and-virtualenv/source-repository-pip-and-virtualenv20:54
clarkbcloudnull: cool, I can get uuids for those instances if you think it will help20:54
dmsimardEmilienM: cool20:54
johnsomDo we have a cache URL we should use for this in gate jobs?20:54
clarkbjohnsom: we do I would haev to look at our nodepool element that caches things for devsatck to find the cache path though20:54
dmsimardEmilienM: added a comment20:55
johnsomOk, yeah, we noticed failures getting this url in some of our gate jobs.20:55
greghaynesget-pip must have had some big outage, image building failed due to that too20:55
EmilienMdmsimard: we can't20:56
clarkbjeblair: I am back to a computer post lunch too and can help with nodepool if necessary20:56
EmilienMdmsimard: only zuul v3 will have this feature20:56
dmsimardEmilienM: so reviews to puppet-designate on mitaka would fail ?20:56
dmsimardEmilienM: or no, it just won't have designate in them I guess20:56
EmilienMnope, they'll just run the scenario003 :)20:56
dmsimardok, sure, whatever20:56
EmilienMyes, without designate20:56
greghaynesclarkb: johnsom /home/jenkins/cache/files/get-pip.py20:56
EmilienMwe'll review our layout in Zuul v320:57
*** tqtran_ has joined #openstack-infra20:57
greghaynesso DIB_REPOLOCATION_pip_and_virtualenv /home/jenkins/cache/files/get-pip.py should do the trick20:58
greghayneser, = not <space>20:58
johnsomgreghaynes Cool, I will give that a try20:58
johnsomYep20:58
*** raildo has quit IRC20:58
*** tqtran has quit IRC20:59
cloudnullclarkb: if you have a few UUIDs handy it'd be great to work from those hosts and try and recreate some of those issues.20:59
*** tuannguyen has quit IRC20:59
mordred"Your team doesn't need to worry about DNS, editing records, CNAMEs or ALIAS. Just focus on shipping."20:59
mordredI feel like the industry is spending so much trying to make sure that engineers don't ever have to worry about engineering20:59
*** tuannguyen has joined #openstack-infra20:59
*** ijw has joined #openstack-infra21:00
clarkbcloudnull: http://logs.openstack.org/19/373219/13/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/f0f5d66/console.html#_2016-09-28_20_40_00_282993 looks like we saw similar there too21:00
clarkbcloudnull: ya let me dig up a couple paris of uuids for you21:00
*** Marx314 has joined #openstack-infra21:00
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Enable container job on multinode  https://review.openstack.org/28891521:01
cloudnullthat'd be great ! thanks21:01
*** ijw has quit IRC21:01
*** trown is now known as trown|outtypewww21:01
clarkbcloudnull: 3cb4a28b-9c3c-40d8-9832-72f6757f5b6a and 640ae349-970d-4ac5-9220-c4b72cebd4f721:01
mordredgreghaynes: so - now the real question is - how do I get that env var all they way from a job into the nodepool builder running in that job21:01
clarkbmordred: you put in your your nodepool.yaml21:01
cloudnullthanks clarkb21:01
clarkbmordred: each image has an env section you can set21:01
*** ijw has joined #openstack-infra21:01
mordredclarkb: oh - right21:02
clarkbcloudnull: 9eadc9db-a4d5-4399-bdbd-79097450dfa7 and 3b225e12-0083-470f-b17a-cc0d06ac448421:02
*** annegentle has quit IRC21:03
*** ijw has quit IRC21:03
clarkbcloudnull: this second pair is from the logs I linked you earlier and the first pair from the one I just linked21:03
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache  https://review.openstack.org/37899921:03
*** rtheis has quit IRC21:03
mordredclarkb: like that ^^ ?21:03
clarkbcloudnull: in both cases ssh between instances on their private IPs seems to have timed out/failed. Our security groups for those nodes should be wide open and we allow all traffic between instances in our iptables setup (but I will work on confirming that)21:03
*** pvaneck has quit IRC21:04
*** tqtran_ has quit IRC21:04
*** spzala has joined #openstack-infra21:04
clarkbmordred: ya I think thats it21:04
*** tuannguyen has quit IRC21:04
mordredgreghaynes: also - is there a 'good' way to get a dib build to use a particular ubuntu mirror?21:04
jlvillalclarkb: In regards to MULTI_HOST https://review.openstack.org/#/c/352790/14/devstack-vm-gate.sh21:04
clarkbmordred: you might want to conditionally add it if the file exists21:04
greghaynesmordred: heh21:04
clarkbmordred: so that the devstack plugin works on not our jenkins hosts21:04
greghaynesmordred: DIB_DISTRIBUTION_MIRROR=foo is what I use, but theres caveats21:04
jlvillalclarkb: How about MULTI_NODE? IRONIC_MULTI_NODE? IRONIC_MULTI_HOST?21:04
clarkbif [ -f /path/to/file ] ; then append that line to file21:05
clarkbmordred: ^21:05
*** ijw has joined #openstack-infra21:05
*** ijw has quit IRC21:05
clarkbjlvillal: Thats probably a better question for devstack devs. I just wanted to point out that already has a meaning in devstack and its not what ironic appears to be using it for21:05
*** ijw has joined #openstack-infra21:05
*** annegentle has joined #openstack-infra21:05
jlvillalclarkb: Thanks. I might go with IRONIC_MULTI_NODE. less confusion for people21:06
*** ddieterly[away] is now known as ddieterly21:06
clarkbjlvillal: ya thats pretty explicit, I like it21:06
greghaynesmordred: basically if you want to get fancy with /etc/hosts you need to also use apt-sources21:06
greghaynesmordred: er with /etc/apt/sources.list21:06
*** yamahata has joined #openstack-infra21:06
*** baoli has joined #openstack-infra21:06
*** tqtran has joined #openstack-infra21:06
mordredgreghaynes: what if I just want to use the exact same setup in /etc/apt as my build host has?21:06
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache  https://review.openstack.org/37899921:07
greghaynesmordred: apt-sources lets you point at a file on your build host. If you use sources.d then youre out of luck unless you want to add some features21:07
*** psilvad has quit IRC21:07
clarkbcloudnull: ya our script definitely attempts to add iptables rules for v4 nad v6 to allow all traffic between the two hosts. Now to confirm it on some actual instances21:07
*** e0ne has quit IRC21:07
mordredclarkb: ^^ updated per your suggestion21:07
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case  https://review.openstack.org/35279021:07
*** kzaitsev_mb has quit IRC21:08
mordredgreghaynes: yah - I mostly would just love for the dib build that nodepool does in its functional test to use the per-region mirror21:08
mordredgreghaynes: so I guess DIB_DISTRIBUTION_MIRROR= is probably fine21:08
greghaynesmordred: ah. So I too would love for the dib functests to do that21:08
mordredgreghaynes: \o/21:08
greghaynesmordred: on a call right now but I should link you to some code21:09
mordredwoot21:09
*** mat128 is now known as mat128|afk21:11
*** jkilpatr has joined #openstack-infra21:11
clarkbcloudnull: ya iptables and ip6tables look all happy so I think that we should be able to communicate there. And in fact in the first job logs I linked much earlier today it did work several times before the failure21:12
clarkbmordred: you need to use >> not > :)21:13
*** dprince has quit IRC21:14
*** ldnunes has quit IRC21:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120921:15
*** tuannguyen has joined #openstack-infra21:15
jeblairclarkb: cool.  i'll begin work on restarting nodepool now21:16
mordredclarkb: hahahahaha21:16
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache  https://review.openstack.org/37899921:16
*** dprince has joined #openstack-infra21:17
mordredclarkb: yay for code review21:17
*** amitgandhinz has quit IRC21:17
*** tiswanso has quit IRC21:19
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for upstart  https://review.openstack.org/37898521:19
*** krtaylor has quit IRC21:21
clarkbpabelanger: is the input confirmation thing related to files existing? making sure I understand ^21:21
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate: Update localrc for ironic-multinode case  https://review.openstack.org/35279021:21
clarkbput another way why is systemd ok?21:21
pabelangerclarkb: ya, there is still a race condition between between when check if the files doesn't exist and when we actually run ssh-keygen.  I am hoping yes n | ssh-keygen works in this instance21:22
*** roxanaghe has joined #openstack-infra21:22
pabelangerclarkb: otherwise, I have no idea how to fix this, aside from removing it from glean21:22
pabelangerthat is what we are fighting right now21:22
pabelangerclarkb: I am not sure why systemd works, I think it is because glean runs much later in the boot process21:23
pabelangerbut that is a guess21:23
*** eharney has quit IRC21:23
*** tonytan4ever has joined #openstack-infra21:23
clarkbpabelanger: also is the yes n required?21:24
clarkbI guess the prompt is for "overwrite file?"21:24
pabelangerclarkb: right, here is the console log of the failure: http://paste.openstack.org/show/583393/21:25
*** dprince has quit IRC21:25
*** edmondsw has quit IRC21:25
*** kzaitsev_mb has joined #openstack-infra21:25
*** pots has joined #openstack-infra21:25
jeblairi have a nodepool launcher and deleter running21:26
clarkbnow we watch and wait to see if it helps?21:27
*** sdague has quit IRC21:27
*** Goneri has joined #openstack-infra21:28
*** tonytan4ever has quit IRC21:28
*** hogepodge has joined #openstack-infra21:28
jeblairclarkb: yeah, unfortunately we will have lost all the events that would have cleared out the 'ready' nodes21:29
jeblairso i'm deleting ones that are 'ready' for more than 1h to speed things up a bit21:29
jeblairwill do the same for 'used'21:30
pabelangeris it possible STATSD_HOST wasn't exported for the new launcher / deleter? I have't seen graphite.o.o update yet21:30
jeblairpabelanger: very possible, i'm just running them from screen experimentally21:31
pabelangerokay cool21:31
openstackgerritMerged openstack-infra/elastic-recheck: Add support for extra elastic-search graph filter  https://review.openstack.org/25558521:31
*** matt-borland has quit IRC21:32
*** Goneri has quit IRC21:33
jeblairpabelanger: restarted with statsd21:33
*** amitgandhinz has joined #openstack-infra21:34
pabelangerclarkb: cool, I think I have it working: http://paste.openstack.org/show/583394/21:34
pabelangerjeblair: thanks21:34
*** yamahata has quit IRC21:35
pabelangerWoah, 935 'in use' nodes21:35
pabelangerthat's new :)21:35
*** jheroux has quit IRC21:36
pabelangerclarkb: going to collect some data to see if it is an issue with systemd too, but I haven't found any failures yet21:38
clarkbok21:38
openstackgerritMerged openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries  https://review.openstack.org/37821521:40
*** Goneri has joined #openstack-infra21:41
*** kzaitsev_mb has quit IRC21:41
*** rwsu has quit IRC21:43
*** kgiusti has left #openstack-infra21:44
*** mriedem has quit IRC21:44
*** spzala has quit IRC21:44
*** EricGonczer_ has quit IRC21:47
*** marst has quit IRC21:50
*** rhallisey has quit IRC21:51
*** mdrabe has quit IRC21:51
*** mdrabe has joined #openstack-infra21:51
jeblairclarkb, pabelanger, fungi: the 2 nodepool processes are taking approx equal cpu time now (i think this is good)21:51
*** tuannguyen has quit IRC21:51
clarkbI concur that that is a good thing21:52
*** tuannguyen has joined #openstack-infra21:52
pabelangeragreed21:52
fungithat's much more of a balance than i anticipated21:52
*** Goneri has quit IRC21:53
*** baoli has quit IRC21:53
pabelangerand we are continuing to grow on used nodes, 1033 now21:54
jeblairnow that they are both cpu bound, we can probably *start* to draw conclusions about how the system is behaving (but we still have a bunch of leaked nodes from the restart)21:54
*** rajinir has quit IRC21:55
jeblairpabelanger: and yeah, that's probably a good sign21:55
fungii suppose if we moved the db into trove we could even divide these between physical servers now?21:56
jeblairfungi: yes, but the host isn't cpu bound yet, so not urgent21:57
*** tuannguyen has quit IRC21:57
*** hasharAway has quit IRC21:57
jeblairfungi: i think i fall on the side of lets do that when we change the launchers to zk21:57
pabelangerclarkb: this just happened with ubuntu-xenial in my testing: http://paste.openstack.org/raw/583397/21:57
jeblairfor now at least :)21:57
pabelangerclarkb: first time I have seen that21:57
fungioh, sure. was mostly pontificating21:57
clarkbpabelanger: is it formatted like that on your end you mean?21:58
pabelangerclarkb: no, that glean is hung on lo0 configuration, and I don't think we have setup a timeout21:58
*** baoli_ has joined #openstack-infra21:58
*** kzaitsev_mb has joined #openstack-infra21:58
pabelangerclarkb: I guess we need to add TimeoutStartSec to systemd21:59
clarkbI thought glean had its own internal timeout22:00
clarkbalso lo should just come up as its logical anyways22:00
pabelangerI'll let the server run and see if it timesout22:00
*** jamesdenton has quit IRC22:01
*** thcipriani has quit IRC22:01
openstackgerritMerged openstack-infra/project-config: Make docs non-voting for quark, remove template  https://review.openstack.org/37790822:02
*** gordc has quit IRC22:02
*** jamesdenton has joined #openstack-infra22:03
*** kzaitsev_mb has quit IRC22:04
*** baoli_ has quit IRC22:04
*** thcipriani has joined #openstack-infra22:04
* mordred boggles at "hung on lo0 configuration"22:05
*** dtardivel has joined #openstack-infra22:05
pabelangerYa, we likely should impose some default timeout, otherwise, systemd keep trying to the end of time22:06
pabelangerA start job is running for Glean fo...erface lo (11min 26s / no limit)22:06
greghaynesoh right, I ran in to that and hacked in a fix and forgot to push it22:07
*** marst has joined #openstack-infra22:07
greghaynesyes put a timeout please22:07
clarkbI mean22:07
greghaynesits because systemd22:08
clarkbdo you want your instance to come up with no lo interface?22:08
mtreinishhaha, I like that >11min to bring up lo22:08
greghaynesif the options are dont come up at all or come up without lo then yes22:08
*** tphummel has quit IRC22:08
clarkbI want to say thats a recipe for broken. Figuring out why it isn't working and correcting that is what we should probably focus on22:08
greghaynesthis also fails in other cases22:08
greghaynesthink dhcp22:08
greghaynesor no connectivity22:09
clarkblo isn't dhcp'd though22:09
clarkband has no connectivity issues22:09
greghaynesright, you need the timeout for the other cases too though22:09
*** mdrabe has quit IRC22:09
greghaynes(which is where I hit it)22:09
*** tqtran has quit IRC22:09
clarkbsure we can timeout the other interfaces22:09
pabelangerYa, I think there is something bigger going on here, something has blocked glean.sh I think22:09
clarkbbut I think lo is a special case that should work 100% of the time22:09
greghaynesright, yea, different bug22:09
pabelangerand is related to ssh host keys I suspect22:09
*** flepied has quit IRC22:10
*** lezbar has quit IRC22:10
pabelangergoing to update 378985 for systemd and test22:10
*** krtaylor has joined #openstack-infra22:10
*** salv-orlando has quit IRC22:10
jeblairpabelanger: the zuul job queue is probably more accurate than nodepool right now (until the leaked nodes are cleared out).  it says we're running 855 jobs.  which is a record.  :)22:11
*** lezbar has joined #openstack-infra22:11
*** tqtran has joined #openstack-infra22:11
*** Goneri has joined #openstack-infra22:11
pabelangercome on 2.1 kilo-jobs per hour22:12
openstackgerritMerged openstack-infra/system-config: Adding user 'maxwell' to OpenStack ID Production  https://review.openstack.org/37868622:12
*** jamesdenton has quit IRC22:12
pabelangerjeblair: Ya, that's the most blue I have see in the 'test nodes' graph on status.o.o/zuul22:13
pabelangerexciting22:13
*** inc0 has joined #openstack-infra22:13
*** thorst has quit IRC22:13
*** baoli has joined #openstack-infra22:13
*** thorst has joined #openstack-infra22:14
*** kzaitsev_mb has joined #openstack-infra22:14
*** rfolco_ has joined #openstack-infra22:14
*** annegentle has quit IRC22:14
*** rfolco_ has quit IRC22:14
*** baoli has quit IRC22:16
*** megm has quit IRC22:16
greghaynesmordred: so last time I hacked on using the per-env mirrors I created https://review.openstack.org/#/c/279691/22:17
*** thorst has quit IRC22:18
greghaynesmordred: which is only for trusty but the idea is we can make an element which has all the logic for the various distros22:18
*** Guest92 has quit IRC22:18
*** esberglu has quit IRC22:19
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts  https://review.openstack.org/37898522:19
*** megm has joined #openstack-infra22:19
greghaynesmordred: I suspect youll run in to that exact issue though - the fact that we use a lot of the same env vars regardless of distro means you cant simply set a bunch of env vars and expect them to all DTRT22:20
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: php5-fpm 503 errors  https://review.openstack.org/37902222:20
*** kzaitsev_mb has quit IRC22:21
*** amitgandhinz has quit IRC22:22
*** inc0 has quit IRC22:23
*** rlandy has quit IRC22:23
*** tonytan4ever has joined #openstack-infra22:23
clarkbcloudnull: any luck with those uuids? let me know if you need a larger sample I can go trolling logstash for more instances22:24
mordredgreghaynes: there are times when I wish dib config wasn't all through env vars22:26
greghaynesmordred: yeeeep22:26
greghaynesmordred: so I have a thing on my todo list22:26
greghaynesmordred: I want dib to accept a yaml file which is basically the diskimages: portion of nodepool.yaml22:26
greghaynesand you can say dib-build image-name22:26
greghayneswhich alleviates some of the env var pain22:26
jeblairgreghaynes: yes please and thank you22:26
mordred++22:27
mordredthat would be ossum22:27
greghaynes:)22:27
*** Apoorva_ has quit IRC22:27
mordredthat said - you know 12-factor apps are all configured via env vars, yeah?22:27
pabelangerI think python-tripleoclient wrote there own yaml to dib thing recently, see it in passing22:27
mordredhttps://12factor.net/config22:27
greghayneshaha22:27
greghaynesmaybe we should all just use djb's envdir22:28
*** Apoorva has joined #openstack-infra22:28
greghaynes(not really)22:28
mordredfactor 3 of 12 factor is the thing that makes me know that it's a pile of utter hogwash22:29
mordred"don't use config management, just write env vars"22:29
greghaynesits one of those things that sounds great until you actually do it with any sizeable app22:29
mordredyup22:30
jesusaurthe underlying principle is still sound though: don't put deployment config in the same repo as the code22:31
mordredyes!22:32
jesusauryou just need to get fancier about where you put it22:32
mordredlike- the part of the description that talks about not hard-coding config into source code ...22:32
johnsomgreghaynes /home/jenkins/cache/files/get-pip.py wasn't found on the host: http://logs.openstack.org/01/379001/1/check/gate-neutron-lbaasv2-dsvm-scenario/dd421fe/console.html#_2016-09-28_22_05_47_90289822:32
mordredthat's very valid - and also I think makes it clear about the audience for the doc22:32
jesusaurya22:32
clarkbbiggest gerrit web ui peeve right now is this focus on the patchset22:32
clarkbwhen someone pushes a new patchset and I hit f5 I want to see the newpatchset!22:33
phschwartzclarkb: +10022:33
clarkbbut no, I have to manually change patchsets otherwise I get the old one again22:33
*** thorst has joined #openstack-infra22:33
greghaynesjohnsom: hrmmm22:33
phschwartzThat is very annoying.22:33
clarkbjohnsom: greghaynes I think devstack-gate may copy it into devstack's files dir if using devstack22:34
greghaynesclarkb: any thoughts on what johnsom ran in to, I thought I was reading the element correctly...22:34
clarkbmordred: ^ you may need to account for that too22:34
greghaynesah22:34
clarkbso its in /opt/stack/new/devstack/files I think22:34
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts  https://review.openstack.org/37898522:34
mordredclarkb: ah22:34
*** esberglu has joined #openstack-infra22:34
johnsommaybe /opt/stack/new/devstack/files/get-pip.py ?22:35
*** cardeois has quit IRC22:36
*** trukise has quit IRC22:36
*** annegentle has joined #openstack-infra22:36
*** annegentle has quit IRC22:36
*** markvoelker has quit IRC22:36
clarkbjohnsom: yes I think so but check if thats where the devstack files dir is relative to devstack22:36
clarkbthat path should be correct for the devstack repo itself22:36
*** esberglu has quit IRC22:37
*** thorst has quit IRC22:38
clarkbyup its devstack/files22:38
*** thorst has joined #openstack-infra22:38
cloudnullClarkb sadly it's just been one of those days where I've been dragged away from computer. I'm currently relocating home and will be digging in there.22:41
*** kzaitsev_mb has joined #openstack-infra22:42
cloudnullSorry for the delay.22:42
clarkbcloudnull: no problem. I am having similar issues with sick children22:42
*** thorst has quit IRC22:43
*** tphummel has joined #openstack-infra22:44
*** ijw has quit IRC22:46
*** thorst has joined #openstack-infra22:48
pabelangerclarkb: fungi: greghaynes: https://review.openstack.org/#/c/378985 is working properly now, updates to ssh-keygen22:48
*** salv-orlando has joined #openstack-infra22:48
pabelangerboth under upstart and systemd22:48
clarkbjeblair: I deleted two 3 hour old nodepool instances in a used state. Checked their zuul launcher and they had both finished their jobs22:50
clarkbthe next oldest instances are less than 1.5 hours old so just gonna let those age a bit more to simplify knowing what is and isn't safe to delete22:51
*** tonytan4ever has quit IRC22:51
*** nadya has joined #openstack-infra22:52
*** salv-orlando has quit IRC22:52
*** thorst has quit IRC22:52
fungismarcet: have you ruled out the possibility that your 503 errors are due to apache not being tuned to handle the request volume you're getting?22:53
*** Swami has quit IRC22:53
fungismarcet: jpmaxman: or are the apache-side errors just these "AH01075: Error dispatching request" and "AH01067: Failed to read FastCGI header" errors i'm seeing from proxy_fcgi in the log?22:54
clarkbrcarrillocruz: for when you wake you might want to respond to this UX survey results thing about quota management and plug your cloud launcher work. I think it would solve some of the problems they describe very well22:55
*** sdake has quit IRC22:56
*** nadya has quit IRC22:57
fungismarcet: jpmaxman: oh, i do see some 503 errors in the apache access log, just not in the error log for some strange reason22:58
*** ijw has joined #openstack-infra22:58
*** ddieterly has quit IRC22:59
jeblairclarkb: yeah, we're at things newer than 1:20 are from the current invocation, so pretty close to that 1.5 hour mark22:59
fungismarcet: jpmaxman: aha, error log simply doesn't mention the http response code. i see now correlating that a lot of them are for the same client ip (v6!) address23:00
jeblairclarkb: no ready nodes, which was the sign of a processing backlog.  however, we have now worked through the zuul backlog, so things should be slowing down now23:01
clarkbjeblair: ya I think its caught up with work demands and so stuff is less pressurized23:01
jeblairclarkb, fungi, pabelanger: i *think* this worked well enough that it's worth writing the init scripts for real23:01
pabelanger++23:01
fungijeblair: it's looking great so far23:01
fungismarcet: jpmaxman: a quick grep of the current log shows 27 out of 31 of these 503 errors were associated with the same ipv6 address, 2 were with another v6 address in the same network, and 2 were from an ipv4 address23:03
fungiis that a load tester maybe?23:03
*** tiswanso has joined #openstack-infra23:03
*** ijw has quit IRC23:03
*** smarcet has quit IRC23:03
*** tiswanso has quit IRC23:03
clarkbfungi: does dns offer any insight?23:04
*** tiswanso has joined #openstack-infra23:04
*** rwsu has joined #openstack-infra23:04
fungiclarkb: no reverse dns, but the v6 addresses are all assigned to rackspace and the v4 address is at some provider in poland23:05
pabelangerjeblair: I can start work in init scripts in the morning23:05
jeblairpabelanger: that would be awesome, thanks!23:06
*** hongbin has quit IRC23:10
*** rbrndt has quit IRC23:12
*** tqtran_ has joined #openstack-infra23:13
*** tqtran has quit IRC23:14
jeblairSep 28 23:10:28 zl01 puppet-user[17540]: Could not find data item zuul_launcher_keytab in any Hiera data file and no default supplied at /opt/system-config/23:15
jeblairproduction/manifests/site.pp:923 on node zl01.openstack.org23:15
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts  https://review.openstack.org/37898523:16
clarkbcloudnull: looking at job logs for taht specific job we hae 44 successes none in osic. and 152 failures mostly in osic23:17
*** kzaitsev_mb has quit IRC23:17
clarkbgiven our instance distribution I would expect most failures and most successes in osic if this wasn't in some way related to the cloud/region23:17
clarkbbut now I know that in general the setup can work. Just a matter of digging into why it isn't happy currently23:18
jeblairpabelanger, ianw: ^ it looks like the keytab is in a zuul-launcher hiera group, but the launchers are in the merger group23:18
*** tqtran_ has quit IRC23:18
openstackgerritMonty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook  https://review.openstack.org/37173423:18
openstackgerritMonty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files  https://review.openstack.org/36960523:18
openstackgerritMonty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async  https://review.openstack.org/37173323:18
jeblairmaybe it just needs to be moved in hiera23:19
*** ijw has joined #openstack-infra23:19
pabelangerjeblair: ah, I think set that up.  Let me check what I did for puppet23:19
pabelangerYa, I think we can just move it in hiera23:20
jeblairpabelanger: ok done :)23:20
jeblaircat zuul-launcher.yaml >>zuul-merger.yaml23:21
jeblairpabelanger: i just did that23:21
jeblairthen deleted it23:21
pabelangerack23:21
*** Goneri has quit IRC23:22
*** kzaitsev_mb has joined #openstack-infra23:22
*** ijw has quit IRC23:24
*** ijw has joined #openstack-infra23:29
*** sdake has joined #openstack-infra23:32
*** ijw has quit IRC23:34
*** sdake_ has joined #openstack-infra23:35
*** markvoelker has joined #openstack-infra23:37
*** hichihara has joined #openstack-infra23:37
*** sdake has quit IRC23:38
*** markvoelker has quit IRC23:41
jeblairyay *now* we have jobs with afs stanzas23:42
jeblairnow we need to merge a change to an infra repo :)23:42
*** kzaitsev_mb has quit IRC23:43
*** kzaitsev_mb has joined #openstack-infra23:44
*** tiswanso has quit IRC23:45
*** rwsu has quit IRC23:45
*** zhurong has joined #openstack-infra23:46
*** ijw has joined #openstack-infra23:47
mordred\o/23:47
mordredjeblair: got a link handy?23:48
jeblairmordred: to what?23:48
pabelangerrcarrillocruz: when you are online, it looks like the nodepool credentials for infracloud-chocolate are not correct23:48
mordredjeblair: oh - I misread your statement - now I understand it23:48
mordred:)23:49
*** xarses has quit IRC23:51
pabelangerrcarrillocruz: I see the issue, I think, typo in hieradata23:51
pabelangerfixing now23:52
*** kzaitsev_mb has quit IRC23:52
*** amitgandhinz has joined #openstack-infra23:53
openstackgerritMonty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook  https://review.openstack.org/37173423:53
openstackgerritMonty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files  https://review.openstack.org/36960523:53
openstackgerritMonty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async  https://review.openstack.org/37173323:53
*** ijw has quit IRC23:53
*** xyang1 has quit IRC23:54
openstackgerritMerged openstack-infra/system-config: Temporarily block port 80 and port 8080 on firehose  https://review.openstack.org/37806023:55
jeblairooh a patch merged!23:56
*** amitgandhinz has quit IRC23:58
*** roxanaghe has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!