| mnaser | got it, thanks :) | 00:00 |
|---|---|---|
| mtreinish | pleia2: ^^^ finally merged | 00:00 |
| anteaya | thank you | 00:00 |
| anteaya | no idea when I will be in montreal next | 00:00 |
| anteaya | but now you are in my address book :) | 00:00 |
| mtreinish | once that gets applied we can run it for a little while and see if it gives us more clues on what's going on | 00:00 |
| *** yamahata has joined #openstack-infra | 00:00 | |
| pleia2 | mtreinish: yay, let me fire this up again | 00:01 |
| pleia2 | (once it gets to the server, that is) | 00:01 |
| anteaya | mnaser: out of curiousity, how many vms do you think vexxhost will donate when we are at quota? | 00:02 |
| mnaser | tough question to answer but the answer is as much as we can, but initially we'd like to grow it up to 100 and see how things are working | 00:03 |
| EmilienM | anteaya: you have to ping me if you're in MTL | 00:03 |
| mnaser | i recently discovered an interesting cool local montreal openstack meetup | 00:04 |
| anteaya | EmilienM: I will indeed | 00:04 |
| *** sflanigan has quit IRC | 00:05 | |
| *** pahuang has quit IRC | 00:05 | |
| *** camunoz has quit IRC | 00:05 | |
| *** aeng has quit IRC | 00:05 | |
| anteaya | mnaser: fair enough, and it might not have been fair of me to ask so sorry to put you on the spot | 00:05 |
| anteaya | mnaser: nice | 00:05 |
| anteaya | do share a report after you attend an event | 00:05 |
| mnaser | http://montrealopenstack.org/en/ -- cool group of people, so try to see if you can drop by :) | 00:05 |
| openstackgerrit | Grant Murphy proposed openstack-infra/project-config: Add bandit gate job for Octavia https://review.openstack.org/286912 | 00:05 |
| mnaser | i attended the most recent one, met some cool people with interesting ideas. dmsimard was there too to talk rdo :) | 00:06 |
| mnaser | i'll have to say he tried to convince me to bring resources so credit goes to him for bringing up the idea again :p | 00:06 |
| clarkb | mnaser: ok I think I have a quick canary tempest run going which should give us an idea of relative performance (which I don't expect to be a real problem) | 00:06 |
| *** ybathia has joined #openstack-infra | 00:06 | |
| *** sridhar_ram has quit IRC | 00:06 | |
| mnaser | nice, hopefully it runs quickly :) | 00:07 |
| EmilienM | mnaser: you in mtl? nice, i lived there before - in qc now | 00:07 |
| mtreinish | clarkb: oh, you just reminded me I need to work on adding the 2nd page performance graphs to o-h (once we get the collection working again) | 00:07 |
| *** annegentl_ has quit IRC | 00:08 | |
| clarkb | mtreinish: sorry I havne't been ignoring your subunit things just mostly head down on trying to add clouds | 00:08 |
| clarkb | milestones and all that | 00:08 |
| mnaser | EmilienM: yep! been here for most of my life, nothing but winter and cold complaints :P | 00:08 |
| mtreinish | clarkb: no worries, I realize that there are other priorities right now | 00:09 |
| *** sridhar_ram has joined #openstack-infra | 00:09 | |
| clarkb | mtreinish: that and taxes | 00:09 |
| mtreinish | I honestly had forgotten about your request for run_time graphs on the 2nd page though | 00:09 |
| clarkb | I got a discount for doing them a month and a half early | 00:09 |
| pleia2 | mtreinish: ok, new subunit-gearman-worker.py is in place, I'm going to start it now and collect some logs | 00:09 |
| mtreinish | oh, and damn my taxes. I need to do that soon too | 00:09 |
| mtreinish | pleia2: cool, thanks | 00:10 |
| smarcet | fungi: its taking its time the fix to be merged | 00:11 |
| *** yamahata has quit IRC | 00:11 | |
| dims | mtreinish : oh man it's Mar 1 already | 00:12 |
| fungi | smarcet: yes, all but one job completed, but gate-infra-puppet-apply-bare-trusty is taking longer than usual | 00:12 |
| *** ashleighfarnham has quit IRC | 00:12 | |
| smarcet | fungi: thx for info :) | 00:12 |
| clarkb | I need to pick the don't clone all the things work back up again in d-g | 00:12 |
| mtreinish | pleia2: well it looks like we added 1 run and no others, so I'm guessing there is a nice stack trace in there already :) | 00:13 |
| clarkb | tripleoci? ya that is setup for every d-g job | 00:13 |
| clarkb | would be an easy one to split off like grenade | 00:13 |
| fungi | clarkb: do the repeated loops through /usr/zuul-swift-logs-env in https://jenkins05.openstack.org/job/gate-infra-puppet-apply-bare-trusty/826/console look normal to you? | 00:13 |
| pleia2 | mtreinish: just stopped it, grabbing log | 00:14 |
| clarkb | fungi: ya I think its a new connection for each file | 00:14 |
| clarkb | fungi: so we get the warning over and over and over | 00:14 |
| fungi | clarkb: oh, so this is all it trying to upload logs after completing the job? | 00:15 |
| fungi | it's taking, like, forever | 00:15 |
| clarkb | yup | 00:15 |
| fungi | and from rax-dfw, which is where the swift container is being hosted | 00:15 |
| pleia2 | mtreinish: erg, not sure how helpful this is http://paste.openstack.org/show/488878/ (I'll email the full log too) | 00:15 |
| fungi | so extra crazy on the crazy | 00:15 |
| fungi | i wonder if rackspace's swift is struggling? | 00:16 |
| *** pahuang has joined #openstack-infra | 00:17 | |
| mtreinish | pleia2: well that's the extra logging I was looking for, between here: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py#n187 | 00:17 |
| mtreinish | and here: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py#n195 | 00:17 |
| mtreinish | subunit_v2 is getting closed | 00:17 |
| mtreinish | still have no clue why, but it narrows it down quite a bit | 00:17 |
| pleia2 | oh good, I'm glad it means something to you :) | 00:17 |
| clarkb | pleia2: nibalizer: want to review the vexxhost creds change? | 00:18 |
| *** zz_dimtruck is now known as dimtruck | 00:18 | |
| pleia2 | sure | 00:18 |
| mtreinish | pleia2: also looking at the full log I can confirm we're not reusing any IO objects (the addrs are all different) | 00:19 |
| pleia2 | +2, but I'll leave for a couple minutes in case nibalizer wants to look too | 00:19 |
| clarkb | pleia2: danke | 00:19 |
| mtreinish | but, I still ahve no idea why it's a closed object when we process it | 00:20 |
| fungi | i may kill this job and see if it completes any faster on another worker | 00:20 |
| fungi | i'm not sure how many more files it has to upload | 00:20 |
| nibalizer | clarkb: sure link? | 00:21 |
| clarkb | nibalizer: https://review.openstack.org/286895 | 00:21 |
| nibalizer | what kind of quota did they give us? | 00:21 |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| clarkb | nibalizer: mnaser wants to start smaller like 25 test instance then ramp up from there | 00:21 |
| nibalizer | cool | 00:21 |
| nibalizer | thanks vexxhost! | 00:21 |
| clarkb | mnaser: sounds like you are ok with the eventual goal of 100 just want to not do it all at once and break things | 00:22 |
| clarkb | mnaser: which is perfectly fine by me | 00:22 |
| nibalizer | clarkb: im realizing that we should have a contacts file | 00:22 |
| nibalizer | so that we know who to poke for what | 00:22 |
| *** aeng has joined #openstack-infra | 00:22 | |
| *** camunoz has joined #openstack-infra | 00:22 | |
| *** aeng has quit IRC | 00:22 | |
| clarkb | nibalizer: if you peruse the passwords file I have done this | 00:22 |
| fungi | nibalizer: well, it's in the passwords list | 00:22 |
| clarkb | nibalizer: :) | 00:22 |
| nibalizer | oh nice | 00:22 |
| fungi | yeah, that | 00:22 |
| nibalizer | ty | 00:22 |
| anteaya | clarkb: is there a project config patch up yet? | 00:22 |
| smarcet | fungi: build failed | 00:22 |
| mnaser | np :) and yep, that's the eventual target | 00:22 |
| smarcet | Patch Set 2: Verified-2 | 00:22 |
| *** aeng has joined #openstack-infra | 00:22 | |
| smarcet | Build failed (gate pipeline). F | 00:22 |
| clarkb | anteaya: not yet, I want to get the base thing in and do initial testing then add the nodepool change in project-config | 00:23 |
| fungi | smarcet: i know, i killed that job and am sending it back in to hopefully get a faster one | 00:23 |
| smarcet | oh ok | 00:23 |
| smarcet | got it :) | 00:23 |
| nibalizer | clarkb: lgtm | 00:23 |
| anteaya | clarkb: sounds good | 00:24 |
| *** gildub has joined #openstack-infra | 00:25 | |
| *** Sukhdev has quit IRC | 00:25 | |
| fungi | smarcet: i was hoping it would just rerun the job when i killed the worker, but in this case it unfortunately marked the job as failed. still should go back through pretty quickly this time unless we get really, really unlucky | 00:25 |
| fungi | yeah, i just watched another change running the same job upload its files in a few seconds | 00:27 |
| fungi | so something was definitely wrong with the worker where we ended up before | 00:27 |
| smarcet | fungi: got it | 00:27 |
| fungi | zuul says estimated time to merge 286896,2 is now 3 minutes | 00:29 |
| fungi | so should hopefully end up in the 00:45 puppet pulse out to the server | 00:29 |
| openstackgerrit | Merged openstack-infra/system-config: Add vexxhost cloud credentials https://review.openstack.org/286895 | 00:30 |
| mnaser | btw, clarkb, you'll notice that right now some servers are not running at full speed (processor speed). that's being resolved tonight so cpu speed for those affected will go from 1.6 to 2.93 per core | 00:30 |
| mnaser | which will probably halve the runtime | 00:30 |
| *** thorst_ has joined #openstack-infra | 00:30 | |
| *** AndyU has quit IRC | 00:31 | |
| mnaser | processor clock speed that is | 00:31 |
| *** ianw has joined #openstack-infra | 00:32 | |
| pabelanger | \o/ canadians | 00:34 |
| ianw | linode grumble "maintenance" grumble "everything's fine status page" | 00:34 |
| fungi | smarcet: okay, this is absurd. we've somehow hit another worker taking forever to upload files after the job succeeds. i'm going to go ahead and merge this change manually so it's not delayed further | 00:34 |
| *** Qiming has quit IRC | 00:34 | |
| smarcet | fungi: thx | 00:34 |
| *** piet has joined #openstack-infra | 00:34 | |
| anteaya | pabelanger: hehehe | 00:35 |
| openstackgerrit | Merged openstack-infra/system-config: OpenstackId relase 1.0.13 https://review.openstack.org/286896 | 00:36 |
| fungi | smarcet: ^ there we go | 00:36 |
| ianw | clarkb: i'm new to these many layers. with your change, i do see the swift client object change between uploads. however, the x-auth-token does *not* seem to change. i think it's effectively cached by os-client-config? unless we create another client-object | 00:36 |
| *** erlon has quit IRC | 00:36 | |
| fungi | jhesketh: any guesses why i've seen two gate-infra-puppet-apply-bare-trusty jobs running in rax-dfw take seemingly forever to upload logs to swift? | 00:37 |
| smarcet | fungi: thx a lot for all your help :) | 00:37 |
| *** keedya has joined #openstack-infra | 00:37 | |
| *** rockyg has quit IRC | 00:37 | |
| fungi | jhesketh: current example (still running) is https://jenkins04.openstack.org/job/gate-infra-puppet-apply-bare-trusty/816/console | 00:37 |
| anteaya | I'm out first thing tomorrow morning, back late morning | 00:37 |
| * jhesketh looks | 00:37 | |
| *** rbrndt has joined #openstack-infra | 00:37 | |
| *** thorst_ has quit IRC | 00:38 | |
| fungi | jhesketh: i watched one in another rax region upload logs in a few seconds | 00:38 |
| fungi | same job | 00:38 |
| clarkb | ianw: I think oscc "caches" it but swiftclient detects that it is old and will try to make a new one that it uses | 00:38 |
| *** thorst_ has joined #openstack-infra | 00:38 | |
| *** derekh has quit IRC | 00:38 | |
| fungi | and, amusingly, this is taking forever in the region where we're actually hosting the swift container | 00:38 |
| jhesketh | fungi: yeah, nothing obvious is jumping out at me... | 00:39 |
| jhesketh | other jobs seem to be uploading okay | 00:39 |
| ianw | clarkb: with what i was doing before, uploading two images back to back, wouldn't you expect to see a new token used between them? | 00:39 |
| ianw | s/used/issued/ | 00:39 |
| fungi | jhesketh: i can ssh into that client and see if, say, there's lots of packet loss or something | 00:39 |
| jhesketh | fungi: the number of files could be part of it | 00:40 |
| jhesketh | possibly a good idea | 00:40 |
| clarkb | ianw: only if the first token had expired | 00:40 |
| fungi | jhesketh: well, as i said, saw the exact same job go quickly in another rax region, so same number of log files | 00:40 |
| *** thorst_ has quit IRC | 00:40 | |
| *** thorst_ has joined #openstack-infra | 00:41 | |
| ianw | clarkb: but i thought the issue was swiftclient can't reissue tokens? | 00:41 |
| clarkb | ianw: it can now that oscc has been updated to pass all the uath info it has through | 00:41 |
| clarkb | ianw: let me go grab the traceback paste again | 00:41 |
| jhesketh | fungi: right, but we've seen 2 gate-infra-puppet-apply-bare-trusty take a while and we haven't experienced this on other jobs? | 00:41 |
| fungi | jhesketh: is there a particular swift endpoint hostname i should ping? | 00:41 |
| fungi | jhesketh: right, so far my sample size is 2 | 00:41 |
| fungi | and they were both the same job | 00:42 |
| *** sarob has joined #openstack-infra | 00:42 | |
| fungi | 172.99.116.7 is this worker | 00:42 |
| clarkb | ianw: http://paste.openstack.org/show/488685/ that is it trying to reauth | 00:42 |
| jhesketh | fungi: the endpoints are here https://jenkins04.openstack.org/job/gate-infra-puppet-apply-bare-trusty/816/parameters | 00:42 |
| *** keedya has quit IRC | 00:42 | |
| *** keedya has joined #openstack-infra | 00:43 | |
| *** ajmiller has quit IRC | 00:43 | |
| fungi | [Wed Mar 2 00:41:40 2016] iptables dropped: IN=eth0 OUT= MAC=bc:76:4e:06:30:ef:e4:c7:22:6c:6e:41:08:00 SRC=193.105.134.220 DST=172.99.116.7 LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=36025 PROTO=TCP SPT=49975 DPT=8080 WINDOW=1024 RES=0x00 SYN URGP=0 | 00:43 |
| fungi | oh, that's something trying to hit 8080/tcp on the worker | 00:43 |
| fungi | so unrelated | 00:44 |
| jhesketh | fungi: I wonder if we're hitting some rate limiting given the number of concurrent threads we have uploading | 00:44 |
| fungi | no obvious packet loss | 00:45 |
| fungi | yeah, maybe | 00:45 |
| fungi | doesn't immediately look like network performance issues anyway (0% loss, rtt<2ms) | 00:45 |
| jhesketh | 100 write operations per second per container | 00:45 |
| jhesketh | If you reach this rate limit, Cloud Files slows the processing of write requests for the container to 100 write operations per second per container. | 00:46 |
| fungi | ooh, so maybe multiple jobs uploading logs simultaneously | 00:46 |
| *** jlk has quit IRC | 00:46 | |
| jhesketh | maybe, but would we be doing more than that? | 00:46 |
| *** sarob has quit IRC | 00:46 | |
| *** jlk has joined #openstack-infra | 00:46 | |
| *** dimtruck is now known as zz_dimtruck | 00:46 | |
| *** jlk has quit IRC | 00:46 | |
| *** jlk has joined #openstack-infra | 00:46 | |
| fungi | dunno. depends on how many complete at the same time i guess | 00:46 |
| *** annegentl_ has joined #openstack-infra | 00:47 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Use instance variable in ByteStreamToStreamResult init https://review.openstack.org/286925 | 00:47 |
| jhesketh | error 429 is too many requests, so if it fails with that we'll have a good idea | 00:47 |
| *** Swami has quit IRC | 00:48 | |
| mtreinish | pleia2: just for a sanity check which version of subunit2sql is installed on the gearman worker? I'm starting to think the bug is in the lower layers | 00:51 |
| pleia2 | elizabeth@subunit-worker01:~$ /usr/local/bin/subunit2sql --version | 00:53 |
| pleia2 | 1.3.0 | 00:53 |
| *** rossella_s has quit IRC | 00:54 | |
| mtreinish | pleia2: ok, that's the latest | 00:54 |
| *** rossella_s has joined #openstack-infra | 00:54 | |
| fungi | ugh, it looks like we're still beset by git remote timeouts. this one just tripped a gate reset https://jenkins03.openstack.org/job/gate-cinder-python27/3128/console | 00:55 |
| fungi | at least zuul-cloner properly used the cache | 00:55 |
| *** annegentl_ has quit IRC | 00:56 | |
| fungi | https://review.openstack.org/282099 would be awesome to have | 00:58 |
| *** zhurong has joined #openstack-infra | 00:59 | |
| *** smarcet has quit IRC | 00:59 | |
| *** tphummel has quit IRC | 01:00 | |
| *** rbrndt has quit IRC | 01:00 | |
| fungi | jhesketh: so that puppet apply job finally completed successfully, just spent >30 minutes uploading logs when it was done | 01:01 |
| *** rbrndt_ has joined #openstack-infra | 01:02 | |
| *** zz_dimtruck is now known as dimtruck | 01:02 | |
| fungi | job ran for 35 minutes in total, 90% of which was spent on log uploading | 01:02 |
| *** sdake has joined #openstack-infra | 01:03 | |
| *** achanda has quit IRC | 01:03 | |
| *** rbrndt_ has quit IRC | 01:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch use of cStringIO.StringIO to io.BytesIO https://review.openstack.org/286926 | 01:06 |
| jhesketh | fungi: yeah and the other puppet apply is still uploading :-s | 01:06 |
| fungi | oh, you found another one? | 01:07 |
| *** chenli has joined #openstack-infra | 01:09 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Remove unused flush from read_subunit() https://review.openstack.org/286928 | 01:09 |
| *** dchen has joined #openstack-infra | 01:13 | |
| *** tphummel has joined #openstack-infra | 01:13 | |
| *** ybathia has quit IRC | 01:15 | |
| *** sdake has quit IRC | 01:16 | |
| clarkb | fungi: mnaser http://paste.openstack.org/show/488885/ that is tempest's results | 01:19 |
| clarkb | now to figure out how that compares to other clouds | 01:19 |
| *** Qiming has joined #openstack-infra | 01:23 | |
| *** Sukhdev has joined #openstack-infra | 01:23 | |
| jhesketh | fungi: sorry, missed your question: I was watching https://jenkins02.openstack.org/job/gate-infra-puppet-apply-bare-trusty/705/console but it finished | 01:23 |
| jhesketh | it did, however, also take 30min | 01:23 |
| *** cody-somerville_ has joined #openstack-infra | 01:23 | |
| clarkb | fungi: mnaser http://paste.openstack.org/show/488886/ rax there seems to be on the quick end and ovh in the middle and this is on par wit hovh | 01:23 |
| jhesketh | and also in dfw | 01:23 |
| clarkb | therefore full steam ahead :) | 01:24 |
| *** cody-somerville has quit IRC | 01:24 | |
| clarkb | I am going to attempt adding a mirror nowish | 01:25 |
| clarkb | hrm | 01:25 |
| clarkb | maybe I should check quotas first /me doe sthis | 01:25 |
| *** jamesmcarthur has joined #openstack-infra | 01:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:26 | |
| *** tphummel has quit IRC | 01:26 | |
| *** tphummel has joined #openstack-infra | 01:27 | |
| *** piet has quit IRC | 01:29 | |
| clarkb | once I figure out how to do it | 01:29 |
| clarkb | ok looks like openstackjenkins has a rough instance quota of 25 which is spot on for the initial requst from mnaser. Perfect | 01:31 |
| *** sdake has joined #openstack-infra | 01:31 | |
| *** sdake has quit IRC | 01:33 | |
| * craige prods jeblair if he's still about, for a fresh load of storyboard, at his convenience. | 01:33 | |
| *** sputnik13 has quit IRC | 01:33 | |
| *** _amrith_ is now known as amrith | 01:34 | |
| *** sam_wan has joined #openstack-infra | 01:36 | |
| *** beisner has quit IRC | 01:38 | |
| *** Daisy has joined #openstack-infra | 01:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add get_stack_outputs method https://review.openstack.org/276487 | 01:40 |
| *** sridhar_ram has quit IRC | 01:40 | |
| *** beisner has joined #openstack-infra | 01:40 | |
| *** fawadkhaliq has quit IRC | 01:42 | |
| *** yamamoto has joined #openstack-infra | 01:45 | |
| fungi | craige: i can mysqldump it. where were you grabbing the result from? anywhere in particular? | 01:46 |
| jeblair | fungi, craige: there is a script on the host | 01:46 |
| jeblair | i wrote it so anyone could easily do it | 01:46 |
| fungi | excellent. doing now | 01:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add environment_files to stack_create https://review.openstack.org/277559 | 01:46 |
| fungi | i manually did one for SotK not long ago, not realizing we had it scripted | 01:46 |
| jeblair | make-db-dump.sh | 01:47 |
| craige | ^^ | 01:47 |
| *** tiswanso has joined #openstack-infra | 01:47 | |
| craige | and I hit it from a URL I have bookmakred fungi | 01:47 |
| jeblair | craige: next time ask roots to run "sudo make-db-dump.sh" :) | 01:47 |
| craige | Will do jeblair. I kept forgetting te script name and couldn;t find therepo it lived in | 01:48 |
| *** tiswanso has quit IRC | 01:48 | |
| fungi | seems to be /root/make-db-dump.sh | 01:48 |
| *** apoorvad has quit IRC | 01:48 | |
| *** tiswanso has joined #openstack-infra | 01:48 | |
| fungi | just found it in root's shell history myself | 01:48 |
| jeblair | er yeah; "sudo /root/make-db-dump.sh" then | 01:48 |
| *** tiswanso has quit IRC | 01:48 | |
| *** tiswanso has joined #openstack-infra | 01:49 | |
| fungi | hrm, sudo -H probably | 01:49 |
| *** rhallisey has joined #openstack-infra | 01:49 | |
| *** tiswanso has quit IRC | 01:49 | |
| fungi | ahh, nope, still prompts for a password. i bet the ~root/.my.cnf needs work | 01:49 |
| jhesketh | yeah I thought there would be a script on the host but my hesitancy was not knowing the sensitivity of the data, if it needed scrubbing or the security of distributing it | 01:49 |
| *** sripriya has quit IRC | 01:49 | |
| jeblair | jhesketh: the script scrubs the data | 01:49 |
| fungi | hrm, nope, sudo -H mysql connects fine | 01:49 |
| *** tiswanso has joined #openstack-infra | 01:50 | |
| jhesketh | jeblair: ah okay, I didn't even log in to check | 01:50 |
| craige | Added trhe script to my etherpad jeblair | 01:50 |
| jeblair | fungi: to be fair, i probably ran it with "sudo su -; ./make-db-dump.sh" | 01:50 |
| clarkb | fwiw I am currently working out how to boot this mirror, its first time with my cinder stuff so different and I have to figure out how to specify the az | 01:50 |
| clarkb | but it is in progress! | 01:50 |
| clarkb | jeblair: you said that the 3.16 kernels on 14.04 don't work with afs right? | 01:50 |
| * clarkb finds an image with 3.13 kernel | 01:51 | |
| *** gongysh has joined #openstack-infra | 01:51 | |
| *** achanda has joined #openstack-infra | 01:51 | |
| fungi | oh, it's explicitly prompting with -p | 01:51 |
| jeblair | clarkb: i don't recall the versions; it was the ones with the later archive thingy (which is in the ubuntu server image but not the ubuntu cloud image) | 01:51 |
| *** mtanino has quit IRC | 01:51 | |
| clarkb | jeblair: thanks I think that is 3.16 and normal is 3.13 | 01:52 |
| jeblair | fungi: ah, might be able to drop that then :) | 01:52 |
| jeblair | clarkb: could probably check a working mirror to be sure | 01:52 |
| *** esikachev has joined #openstack-infra | 01:52 | |
| fungi | jeblair: i have made a working version by dropping -u, -p and -h | 01:52 |
| fungi | the original has a .old suffix appended | 01:53 |
| fungi | craige: i've run it now | 01:53 |
| jeblair | clarkb: if you are stuck with an image with the archive thingy, our puppet will remove the archive, and then some about of bashing "apt-get remove" at various kernel things with higher numbers should get it back into a state | 01:53 |
| jeblair | fungi: ++ | 01:53 |
| *** chenli has quit IRC | 01:53 | |
| *** chenli has joined #openstack-infra | 01:54 | |
| *** jaypipes has quit IRC | 01:54 | |
| *** jpr has quit IRC | 01:54 | |
| clarkb | jeblair: I found one. vexxhost does both the .2 .3 .4 .5 images which have new kernels and up to date unpointed release images which have old kernelks | 01:54 |
| *** Daisy has quit IRC | 01:54 | |
| jeblair | oh interesting | 01:54 |
| jeblair | (old, but maintained!, kernels, for the record) | 01:55 |
| *** Daisy has joined #openstack-infra | 01:55 | |
| *** ybathia has joined #openstack-infra | 01:55 | |
| craige | thanks fungi, nabbing it | 01:55 |
| clarkb | now I just need to sort out how to pass the availability zone | 01:55 |
| *** Daisy has quit IRC | 01:55 | |
| clarkb | (the az stuff is documented in the passwd file not sure if there is a better place for that but cinder needs a specific az) | 01:55 |
| *** Daisy has joined #openstack-infra | 01:55 | |
| craige | Got fungi. Thanks fungi jeblair | 01:55 |
| craige | Um, "got it, fungi" | 01:56 |
| clarkb | looks like create_server takes an arg | 01:56 |
| *** thorst_ has quit IRC | 01:56 | |
| *** esikachev has quit IRC | 01:56 | |
| *** thorst_ has joined #openstack-infra | 01:57 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Remove inaccurate comment https://review.openstack.org/286932 | 01:57 |
| mriedem | ianw: can we get a reply on this old patch? https://review.openstack.org/#/c/215929/ | 01:57 |
| *** Sukhdev has quit IRC | 01:58 | |
| fungi | it's my new ad campaign: "got fungi?" | 01:58 |
| fungi | i stole it from an antifungal manufacturer | 01:58 |
| * Clint twitches. | 01:59 | |
| mriedem | ianw: actually, nevermind, we should probably just ignore that one | 01:59 |
| *** pvaneck has quit IRC | 02:00 | |
| *** bpokorny has quit IRC | 02:01 | |
| *** gokrokve has quit IRC | 02:01 | |
| ianw | mriedem: in general, i still prefer to avoid things that override project defaults, because the project changes and devstack never does and then we're not testing what we think we're testing. i'd have to context switch back to that issue though | 02:01 |
| mriedem | ianw: yeah i've lost it more or less, | 02:01 |
| mriedem | it was for an lxc job | 02:02 |
| mriedem | thomas maddox can pick it up if he wants since he's pursuing that job now | 02:02 |
| *** jamesmcarthur has quit IRC | 02:03 | |
| *** ashleighfarnham has joined #openstack-infra | 02:03 | |
| *** kzaitsev_mb has quit IRC | 02:05 | |
| *** thorst_ has quit IRC | 02:05 | |
| *** woodster_ has quit IRC | 02:06 | |
| clarkb | this is the longest launch node command I have ever used but it is doing things | 02:07 |
| fungi | heh | 02:07 |
| fungi | due to the alternate az for the cinder volume? | 02:07 |
| clarkb | and needing to specify all the cinder volume details (label, mount path, etc | 02:07 |
| clarkb | also vexxhost uses 'public' for the public IPs so I thin kexisting nodepool will just work | 02:08 |
| *** rfolco_ has joined #openstack-infra | 02:08 | |
| *** rfolco_ has quit IRC | 02:08 | |
| clarkb | though launch node isn't saying that it has ssh'd in to do things so may be mor ework to do there | 02:09 |
| clarkb | oh! could be a security group thing I only checked the one account | 02:10 |
| clarkb | that is dod | 02:10 |
| mordred | clarkb: did we get vexxhosts? | 02:11 |
| *** rhallisey has quit IRC | 02:11 | |
| fungi | mordred: get, got and gotted | 02:11 |
| mordred | WOOT! | 02:11 |
| * mordred hands mnaser a pony | 02:11 | |
| *** Daisy has quit IRC | 02:12 | |
| fungi | if clarkb doesn't get waylaid by babies soon, we may have vexxhost in nodepool before the night is out | 02:12 |
| * mnaser is literally standing behind the very servers you're using | 02:12 | |
| mnaser | :p | 02:12 |
| mordred | mnaser: nice! | 02:12 |
| *** Daisy has joined #openstack-infra | 02:12 | |
| mnaser | thanks mordred | 02:12 |
| mordred | mnaser: be careful - he only eats at Joe Beef | 02:13 |
| clarkb | mordred: if you stick around we might just get noepool using it tonight | 02:13 |
| mnaser | so the cpu scaling stuff is solved, so these bad boyz are going to go from 1.6ghz per core to 2.93ghz and "turbo" at 3.33ghz | 02:13 |
| clarkb | mnaser: wow nice | 02:14 |
| mordred | ooh! turbo! | 02:14 |
| mnaser | anyone else think its funny how we had hyperthreading and "turbo" in the pentium 4 times | 02:14 |
| mnaser | and now that's all back | 02:14 |
| * fungi remembers that button on his computer | 02:14 | |
| mnaser | :p | 02:14 |
| clarkb | ok I think security groups are fixed now, time to wait for shade launch node to either fail or move on | 02:14 |
| clarkb | (this way I don't have to do manual cleanup) | 02:14 |
| clarkb | I will also need to add a patch for az support to shade launch node | 02:14 |
| mnaser | anyways most of my dc work is done \o/ i'll bbl, if you need anything please let me know | 02:14 |
| clarkb | I mean I wrote it, I need to push it up once it works | 02:14 |
| fungi | all the button did was change the 2x7-segment led on the case to change from 25 to 50 | 02:15 |
| fungi | or if you were lucky, 33 to 66 | 02:15 |
| mordred | I so wanted a 33/66 | 02:15 |
| mnaser | btw, clarkb -- there is much more capacity at ca-ymq-1 but no cinder, less capacity at ca-ymq-2 but faster procs, 10gig internal networking and cinder | 02:15 |
| mnaser | but we're adding more in ca-ymq-2 as we go | 02:15 |
| mnaser | the reason we have less capacity is people just end up using it because it's better, but the tests should be able to run in any az | 02:16 |
| clarkb | mnaser: we will only do the explicit az where we need cinder | 02:16 |
| mnaser | yep thats ideal | 02:16 |
| clarkb | mnaser: whcih is one host today so thats fine | 02:16 |
| fungi | though that does mean all our workers are connecting from az1 to the mirror hosted in az2 | 02:16 |
| mnaser | theres a private line between them | 02:17 |
| clarkb | fungi: I think its mostly logical and not a big deal | 02:17 |
| clarkb | ya that | 02:17 |
| *** Daisy has quit IRC | 02:17 | |
| mnaser | you should get gige speeds | 02:17 |
| fungi | so if there are serious bandwidth bottlenecks (they'd have to be pretty terrible) we might notice | 02:17 |
| clarkb | if that becomes an issue we can just use a bigger flavor for the mirror host and use local disk | 02:17 |
| *** ybathia has quit IRC | 02:17 | |
| fungi | gig-e is way more than sufficient for mirror traffic | 02:17 |
| clarkb | ok time out for the failed ssh is forever I am going to manually cleanup and try again | 02:17 |
| mnaser | also be good if mirrors ran over tenant network ip @ ca-ymq-2 so any workers @ ca-ymq-2 will use 10gig and ca-ymq-1 will be over 1g | 02:17 |
| *** aeng has quit IRC | 02:18 | |
| mnaser | traffic over public runs @ 1gig vs tenant networks @ 10gig | 02:18 |
| *** camunoz has quit IRC | 02:18 | |
| fungi | really the amount of network traffic these generate should be pretty minimal. we try to avoid network use as much as possible in jobs and pre-cache as much as we can on the images | 02:18 |
| mnaser | yep thats ideal | 02:19 |
| fungi | so actually, our image uploads (once a day) are probably more aggregate network transfer than our workers and mirror syncs | 02:19 |
| clarkb | ya its mostly just to get an up to date apt pacakge list and a few small python wheels | 02:19 |
| clarkb | should be small | 02:19 |
| clarkb | (also I may have derped the cinder attachment by getting impatient. that will leanr me | 02:19 |
| mnaser | :P | 02:19 |
| *** sflanigan has quit IRC | 02:19 | |
| mnaser | ill bbl, driving home now | 02:19 |
| fungi | thanks again mnaser! | 02:20 |
| *** keedya_ has joined #openstack-infra | 02:23 | |
| clarkb | mordred: another openstackclient thing, it tells you instance name not uuid for volume attachment | 02:23 |
| *** keedya has quit IRC | 02:23 | |
| clarkb | and instance names need not be unique | 02:24 |
| mordred | clarkb: that's awesome | 02:24 |
| mordred | clarkb: (you know these days most of he time I just fire up a python repl and import shade :) ) | 02:24 |
| clarkb | mordred: I am really trying to use this openstackclient thing because everyone keeps deprecating the old stuff | 02:25 |
| clarkb | mordred: fwiw I think my current thing is hanging on waiting for the volume to attach | 02:25 |
| clarkb | mordred: anything obviously wrong with https://review.openstack.org/#/c/285477/5/launch/shade-launch-node.py ? | 02:26 |
| clarkb | perhaps the az thing is tripping shade up? | 02:26 |
| mordred | clarkb: looking | 02:27 |
| mordred | clarkb: it shouldn't be - but maybe? | 02:27 |
| mordred | I'm pretty sure that's going to be working by id which shouldn't be a problem | 02:27 |
| clarkb | status is in-use so I think its all attached | 02:28 |
| clarkb | but shade-launch-node doesn't seem to be moving forward | 02:28 |
| mordred | clarkb: if you put at the top of the script "shade.simple_logging(debug=True)" (or setup logging module logging) shade will tell you every api call it makes just like nodepool | 02:28 |
| mordred | clarkb: although that won't tell you if it's not seeing the .... OH | 02:28 |
| mordred | clarkb: I think I may know what it is | 02:28 |
| clarkb | mordred: cool I await for magicallness | 02:28 |
| mordred | oh - nope. nevermind. | 02:29 |
| clarkb | boo | 02:29 |
| mordred | you're not running with caching on | 02:29 |
| mordred | I was thinking cache invalidation logic | 02:29 |
| mordred | but nope | 02:29 |
| clarkb | at this point I may just wait for timeout to happen to confirm it is indeed the volume thing | 02:29 |
| *** gokrokve has joined #openstack-infra | 02:29 | |
| mordred | clarkb: actually, shade.simple_logging(debug=True, http_debug=True) will have it log the resuests stuff | 02:29 |
| clarkb | I will ad that for the next run | 02:30 |
| mordred | cool | 02:30 |
| mordred | it'll be chatty and stuff, but should show you if it's the status | 02:30 |
| clarkb | ya | 02:30 |
| *** camunoz has joined #openstack-infra | 02:31 | |
| *** sflanigan has joined #openstack-infra | 02:31 | |
| * Shrews needs a way to disable highlighting on "shade" in the evenings | 02:31 | |
| *** aeng has joined #openstack-infra | 02:32 | |
| ianw | clarkb: i have a theory -> https://etherpad.openstack.org/p/raxuploadissues ; it may be hard to follow | 02:32 |
| clarkb | oh! shade doesn't have a timeout by default it will just go forever so I should kill this and try again with debugging in place | 02:33 |
| *** gokrokve has quit IRC | 02:34 | |
| ianw | i think shade might create a swiftclient.service.SwiftService object via auth_token ... maybe it expires, but then the client goes and tries to _retry when it gets an auth failure, at that point it doesn't have auth info and hits the "i have no tenant" error | 02:35 |
| clarkb | mordred: ya I think it is spinning on volume_attach wait=True | 02:36 |
| *** yamamoto has quit IRC | 02:36 | |
| clarkb | mordred: the volume attach was the last thing it did and now its waiting for 2 seconds over and over | 02:36 |
| clarkb | mordred: my best theory is az required for that op? | 02:36 |
| *** tphummel has joined #openstack-infra | 02:37 | |
| openstackgerrit | Merged openstack-infra/nodepool: Enhance message for image ssh auth https://review.openstack.org/222565 | 02:38 |
| clarkb | mordred: though I am not seeing repeated requests for the volume so there is some weight to your cache theory | 02:39 |
| clarkb | mordred: how do I disable caching? | 02:39 |
| *** sripriya has joined #openstack-infra | 02:39 | |
| *** Daisy has joined #openstack-infra | 02:40 | |
| clarkb | actually I am pretty vconvinced this is the problem | 02:40 |
| *** Daisy has quit IRC | 02:42 | |
| *** Daisy has joined #openstack-infra | 02:42 | |
| clarkb | ya thats it, the cache is only invalidated the first time around | 02:43 |
| clarkb | needs to be invalidated each time around | 02:43 |
| clarkb | let me see if I can install a local version and test it and confirm it working then push patch to shade | 02:43 |
| *** gokrokve has joined #openstack-infra | 02:45 | |
| *** mriedem has quit IRC | 02:46 | |
| *** Daisy has quit IRC | 02:47 | |
| *** rguillebert has quit IRC | 02:47 | |
| *** dimtruck is now known as zz_dimtruck | 02:48 | |
| *** jamesmcarthur has joined #openstack-infra | 02:49 | |
| clarkb | mordred: confirmed local patch got past that | 02:49 |
| clarkb | will push in a few | 02:49 |
| *** cody-somerville_ is now known as cody-somerville | 02:52 | |
| *** cody-somerville has joined #openstack-infra | 02:52 | |
| *** rlandy has quit IRC | 02:54 | |
| clarkb | now running into problems with device enumeration | 02:55 |
| *** annegentl_ has joined #openstack-infra | 02:56 | |
| stevemar | clarkb: we're more than happy to see osc fixes :) | 02:58 |
| stevemar | clarkb: but i do appreciate you giving osc an honest try | 02:59 |
| clarkb | stevemar: ya I am trying to use it and more likely to fall back on old clients to get things done currently but if I find time will swing around and write some patches | 03:00 |
| stevemar | clarkb: thats more than most people are willing to do :) | 03:00 |
| clarkb | mordred: so on vexxhost /dev/vdb is your config drive but thats where we try to make swap and that explodes | 03:00 |
| * clarkb makes a hack in make_swap.sh | 03:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:01 | |
| *** annegentl_ has quit IRC | 03:02 | |
| *** thorst_ has joined #openstack-infra | 03:04 | |
| clarkb | and curiously cinder reports it also attached to /dev/vdb | 03:04 |
| clarkb | /dev/vdb is popular tonight | 03:04 |
| *** chenli has quit IRC | 03:05 | |
| clarkb | and confirmed that according to blkid /dev/vdb is actually the config drive | 03:05 |
| clarkb | mnaser: ^ not sure if oyu are still awake but it seems like cinder doesn't agree with the nova instance on what the device name for attached volumes is | 03:06 |
| clarkb | mnaser: I probably won't debug that more tonight since dinner just showed up | 03:06 |
| *** kzaitsev_mb has quit IRC | 03:06 | |
| *** zz_dimtruck is now known as dimtruck | 03:08 | |
| fungi | i've heard tell of auto-attach selecting the wrong block device name, and someone suggested you're better off explicitly feeding it one instead | 03:09 |
| fungi | i have not substantiated this as a viable option, however | 03:09 |
| *** alivigni has joined #openstack-infra | 03:10 | |
| *** thorst_ has quit IRC | 03:10 | |
| clarkb | fungi: the problem with that is you can conflict, but Iguess thats no worse than getting the wrong mapping | 03:10 |
| fungi | lbyl i guess | 03:10 |
| clarkb | mnaser: consider this an informal bug report, /dev/vdb is config drive and /dev/vdc is the attached volume but /dev/vdb is what cinder/nova seem to agree on as the attached volume | 03:11 |
| clarkb | fungi: I can add explicit device support and just hardcode to /dev/vdc which should get past this | 03:11 |
| clarkb | which I am tempted to do, however it seems like something openstack should be handling for me | 03:11 |
| *** gokrokve has quit IRC | 03:12 | |
| fungi | i couldn't agree more | 03:13 |
| fungi | it's a bug for sure | 03:13 |
| *** annegentl_ has joined #openstack-infra | 03:13 | |
| mnaser | hello im here | 03:15 |
| mnaser | that's an openstack bug clarkb | 03:15 |
| mnaser | you want to know the next fun part? | 03:15 |
| mnaser | reboot the machine | 03:15 |
| mnaser | configdrive goes to /dev/vdc | 03:15 |
| *** dims has quit IRC | 03:15 | |
| *** sdake has joined #openstack-infra | 03:15 | |
| mnaser | the device goes to /dev/vdb | 03:15 |
| * mnaser sighs | 03:15 | |
| clarkb | mnaser: wow | 03:16 |
| clarkb | mnaser: thats amazing, | 03:16 |
| clarkb | I will go ahead and add explicit device path support | 03:16 |
| clarkb | thingee: ^ how to fix? | 03:16 |
| mnaser | #justopenstackthings | 03:16 |
| mnaser | clarkb: ask a workaround, do you need configdrive or is 169.254.169.254 sufficent? | 03:16 |
| *** gyee has quit IRC | 03:16 | |
| mnaser | you could add --config-drive=false to your boot command and as far as i know, that'll force it to not boot from configdrive | 03:17 |
| *** dimtruck is now known as zz_dimtruck | 03:18 | |
| clarkb | mnaser: we don't need config drive but generally we prefer it because it is more reliable than the metadata service | 03:18 |
| clarkb | mnaser: I think if I explicitly attach at /dev/vdc it will work fine | 03:18 |
| clarkb | about to test this | 03:18 |
| mnaser | yep, i should file a bug about this in cinder/nova | 03:18 |
| mnaser | ran into this issue with some customers in the past | 03:18 |
| clarkb | ++ please do | 03:18 |
| clarkb | and maybe we can rope thingee into helping with fixing it | 03:18 |
| mnaser | i wonder if its an image problem too | 03:19 |
| *** chenli has joined #openstack-infra | 03:20 | |
| mnaser | aka images+cloud-init should relabel /dev/disk/by-label/config-2 to /dev/vdz or something, ill do some searching | 03:20 |
| * thingee was having dinner, and catching up | 03:21 | |
| clarkb | hrm it seems to be ignoring me when I specify /dev/vdc and using /dev/vdb anyways | 03:21 |
| *** jamesmcarthur has quit IRC | 03:21 | |
| mnaser | clarkb: http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Cinder::VolumeAttachment-prop-mountpoint | 03:21 |
| mnaser | unrelated, but kinda related. | 03:22 |
| clarkb | thingee: tl;dr is / is on /dev/vda, config drive is on /dev/vdb, and an attached cinder volume is at /dev/vdc. but cinder and nova both agree that the volume is attached at /dev/vdb | 03:22 |
| *** weshay has quit IRC | 03:22 | |
| clarkb | thingee: so then you attempt to format it and od things and it explodes because the poor vfat thing that is there for config drive is all sad | 03:22 |
| *** yamamoto has joined #openstack-infra | 03:22 | |
| *** gokrokve has joined #openstack-infra | 03:23 | |
| *** david-lyle has joined #openstack-infra | 03:24 | |
| mnaser | clarkb: volumeid "96507b68-5fdb-47a1-8653-f1c1d7d7eed4" is located at /dev/disk/by-id/virtio-96507b68-5fdb-47a1-8 | 03:25 |
| mnaser | that may be a much more stable way of determining where it is... until cinder+nova stuff are settled | 03:25 |
| clarkb | mnaser: are you retrieving that via a cinder or nova api? | 03:26 |
| clarkb | oh its the uuids I see | 03:26 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Tweak regex for doc-only changes https://review.openstack.org/286956 | 03:26 |
| *** sarob has joined #openstack-infra | 03:26 | |
| mnaser | yeah the cinder volume uuid, not sure if you need it inside the machine (but i figure if you know from advance its /dev/vdb .. you should be able to get the id) | 03:27 |
| clarkb | yup I can get the id from where I am querying | 03:27 |
| clarkb | then use that to determine the device path | 03:27 |
| *** larainema has quit IRC | 03:27 | |
| mnaser | yep, looks like its trimmed only the first 20 characters | 03:27 |
| *** gokrokve has quit IRC | 03:27 | |
| clarkb | I wonder if that is consistent across cloud sthough | 03:28 |
| mnaser | im checking where that's coming from right now, clarkb :) im curious | 03:29 |
| clarkb | mnaser: also is /dev/disk/by-uuid not going to have the cinder uuid I bet | 03:29 |
| clarkb | mnaser: because that would be even easier | 03:29 |
| mnaser | it doesnt, i checked :( | 03:29 |
| mnaser | "in a world..." | 03:29 |
| *** larainema has joined #openstack-infra | 03:29 | |
| clarkb | ok so that might be a good feature request :) | 03:30 |
| *** annegentl_ has quit IRC | 03:30 | |
| clarkb | I think I found the bug in my explicit device path request, if this works I will go with it for now | 03:30 |
| *** sarob has quit IRC | 03:31 | |
| *** camunoz has quit IRC | 03:32 | |
| *** sflanigan has quit IRC | 03:33 | |
| clarkb | that didn't work, it appears that cinder/nova are really convinced that it is mounted at /dev/vdc | 03:33 |
| clarkb | er vdb | 03:33 |
| thingee | clarkb: so cinder only cares about /dev/disk ,not what the vm has on /dev/sdb | 03:33 |
| *** zz_dimtruck is now known as dimtruck | 03:33 | |
| *** aeng has quit IRC | 03:33 | |
| *** tphummel has quit IRC | 03:34 | |
| thingee | clarkb: so technically not a consistent thing between cinder and nova. | 03:34 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:34 | |
| clarkb | thingee: but the cinder api reports where it is attached as /dev/vdb | 03:34 |
| clarkb | thingee: so it must care | 03:34 |
| thingee | that's what nova gives it though. :) | 03:34 |
| thingee | cinder is not on the vm | 03:35 |
| fungi | so it's more that nova is making wild accusations about devnames | 03:35 |
| mnaser | nova should know there is a configdrive attached at /dev/vdb though. | 03:35 |
| mnaser | also, i think its supposed to be /dev/disk/by-id/virtio-serial .. where serial is the uuid of the volume, but it seems like it's being trimmed off at 20 characters | 03:36 |
| mnaser | udevadm info /sys/class/block/vdb | grep ID_SERIAL => E: ID_SERIAL=96507b68-5fdb-47a1-8 | 03:36 |
| mnaser | but in the libvirt definition, <serial>..</serial> contains the full uuid | 03:36 |
| thingee | clarkb, mnaser also cryptsetup which is what nova is doing for encryption of block devices has some limit with the device path. I looked into it at one point, but couldn't figure out the code. | 03:36 |
| thingee | clarkb, mnaser need that fixed before we can add uuids to diskpath | 03:36 |
| clarkb | progress! now I fail on pvcreate | 03:37 |
| clarkb | (it isn't installed | 03:37 |
| mnaser | oh heh | 03:38 |
| thingee | when I was working at a block storage vendor, I had to have them trim the iqn down because of the cryptsetup limit :P | 03:38 |
| mnaser | thingee: that sounds awfully fun to do, lol | 03:38 |
| * thingee looks at nova code | 03:38 | |
| mnaser | clarkb: feel free to upload any images. the ones we use might be a bit weird because we use an agent to config network and stuff (but we're moving to qemu-guest-agent soon) | 03:38 |
| *** sdake has quit IRC | 03:39 | |
| mnaser | so if the vanilla ubuntu ones work for you, you can use them too | 03:39 |
| clarkb | ok that is probably my best bet | 03:39 |
| thingee | oh wow, I forgot the number of "ebs" mentions in this code. | 03:41 |
| *** kaisers has quit IRC | 03:42 | |
| *** fawadkhaliq has joined #openstack-infra | 03:43 | |
| clarkb | 'ubuntu-trusty' image upload is in prgress | 03:43 |
| *** kaisers has joined #openstack-infra | 03:44 | |
| *** yuanying has quit IRC | 03:44 | |
| *** amrith is now known as _amrith_ | 03:44 | |
| thingee | clarkb: so the image was deployed by image that has these partitions? | 03:44 |
| fungi | clarkb: this may be our first attempt at using a dib worker image for a long-lived server, so experience could involve new and exciting discoveries | 03:45 |
| fungi | thingee: instance booted with rootfs, nova attaches configdrive as a second block device, calls to cinderclient (via openstackclient i guess?) create third block device which are then attached via novaclient (openstackclient?) calls | 03:46 |
| fungi | that last step, nova is attaching and attempting to use the same device name it already used when adding the configdrive | 03:47 |
| clarkb | fungi: I am using the canonical published image actually since that is apparently what we did in bluebox and infracloud | 03:47 |
| *** dimtruck is now known as zz_dimtruck | 03:47 | |
| fungi | clarkb: oh, so canonical's ubuntu trusty cloud image, not our dib-built "ubuntu-trusty" image | 03:47 |
| clarkb | thingee: right the image doesn't have a vdb or vdc, what fungi describes is what happens | 03:47 |
| clarkb | fungi: ya | 03:47 |
| clarkb | if my 10mbps upload ever finishes (if I was smart I wouldnt do this from home) | 03:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Also reset swift server client https://review.openstack.org/286964 | 03:48 |
| *** annegentl_ has joined #openstack-infra | 03:48 | |
| clarkb | comcast's 300gb per month quota is just a suggestion right? | 03:48 |
| mnaser | clarkb thingee - https://github.com/qemu/qemu/blob/master/hw/block/virtio-blk.c#L561-L569 and https://github.com/qemu/qemu/blob/master/include/standard-headers/linux/virtio_blk.h#L52 | 03:48 |
| ianw | clarkb: ^ so this is my best guess here. essentially a SwiftService object has a client object inside it, so we need to refresh it in the same way | 03:48 |
| mnaser | so qemu truncates serials to first 20 characters by default | 03:49 |
| *** camunoz has joined #openstack-infra | 03:49 | |
| *** sflanigan has joined #openstack-infra | 03:49 | |
| *** sflanigan has joined #openstack-infra | 03:49 | |
| *** aeng has joined #openstack-infra | 03:49 | |
| mnaser | clarkb: just an fyi, the images we use are actually published by ubuntu (we're a certified ubuntu cloud) .. they just have some extra stuff that i know will come in the way for your use case instead of being useful | 03:49 |
| clarkb | mnaser: gotcha | 03:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Also reset swift service object at upload time https://review.openstack.org/286964 | 03:50 |
| mnaser | nova has this poopy limitation where it caches images locally based on uuid | 03:50 |
| mnaser | if it didnt do that, we'd have this thing where we have a "master" uuid that always deploys latest ubuntu release from http url as location in glance | 03:51 |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Also reset swift service object at upload time https://review.openstack.org/286964 | 03:51 |
| mnaser | but that wouldn't work because nova assumes that if uuid doesnt change, the image doesnt change. | 03:51 |
| *** gokrokve has joined #openstack-infra | 03:51 | |
| clarkb | mnaser: apparently pvcreate is no longer included (or maybe was never included) on ubuntu's base images | 03:52 |
| mnaser | i think its not anymore probably because they don't use lvm in the images? | 03:52 |
| clarkb | with that I think I really need to stop working for that day, I have a viable workaround for the device listing thing | 03:52 |
| *** sdake has joined #openstack-infra | 03:53 | |
| clarkb | I just need to figure out how I want to get lvm tooling on the images | 03:53 |
| *** esikachev has joined #openstack-infra | 03:53 | |
| clarkb | mnaser: that would probably explain it | 03:53 |
| fungi | clarkb: do we try to add the volume before puppeting? | 03:53 |
| *** jpr has joined #openstack-infra | 03:53 | |
| fungi | clarkb: otherwise, the lvm2 package seems like something we should be declaring in puppet anyway | 03:53 |
| clarkb | fungi: yes, because puppet slaps stuff down in the volumes | 03:54 |
| fungi | aha | 03:54 |
| fungi | catch-22 | 03:54 |
| fungi | good call, sleep on it | 03:54 |
| clarkb | this is actually an excellent case for buiding our own images specifically for this | 03:54 |
| clarkb | also as an alternative I could just use the bigger flavor which will have bigger local disk and use that instead | 03:55 |
| * fungi is pretty sure our dib-built ubuntu-trusty images have lvm2 installed | 03:55 | |
| clarkb | rather than try to rely on cinder and images and all that working out | 03:55 |
| *** gokrokve has quit IRC | 03:55 | |
| fungi | but also glean, so new unknowns | 03:55 |
| openstackgerrit | Merged openstack/requirements: Bump to final Mitaka python-saharaclient https://review.openstack.org/286503 | 03:57 |
| *** esikachev has quit IRC | 03:57 | |
| thingee | if I had to guess why nova and cinder have /dev/sdb for the attached volume is because of the device being reserved in the second attach not completing when the third volume/create//attach was happening. | 03:58 |
| *** jpr has quit IRC | 03:58 | |
| *** sflanigan has quit IRC | 03:58 | |
| fungi | that would be odd since the configdrive is potentially needed during boot | 03:58 |
| fungi | is our launch-node script attaching the new volume after the instance boots, or adding it in the boot call? | 03:59 |
| clarkb | fungi: it is a separate attach after boot | 04:00 |
| thingee | ah k | 04:00 |
| *** chenli has quit IRC | 04:01 | |
| mnaser | if you created the volume and booted it attached, i suspect you won run into this issue | 04:01 |
| *** zz_dimtruck is now known as dimtruck | 04:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:02 | |
| mnaser | nova boot ... --block-device source=volume,id=<..>,dest=volume,shutdown=preserve ... | 04:03 |
| mnaser | that might prevent the issue from happening | 04:03 |
| *** aeng has quit IRC | 04:04 | |
| thingee | mnaser: isn't that what's happening now since fungi said the config drive is needed for boot? or do you mean attach the third one at boot? | 04:05 |
| *** sam_wan has quit IRC | 04:05 | |
| mnaser | thingee: right now, i believe that a nova boot then nova volume-attach is happening (2 steps) -- what im suggesting is to boot directly with the volume attached | 04:05 |
| mnaser | i suspect this may just boot it with the right order as it will be aware that it has to add the volume | 04:06 |
| clarkb | yes it is two steps right now | 04:06 |
| thingee | mnaser: again clarifying, attach which? config? the third volume? | 04:06 |
| *** yuanying has joined #openstack-infra | 04:07 | |
| thingee | as I understood there are two attaches happening | 04:07 |
| mnaser | configdrive isn't attached, it's always setup as the last device on the instance | 04:07 |
| *** kzaitsev_mb has quit IRC | 04:07 | |
| thingee | I thought fungi said it's needed for boot though | 04:07 |
| mnaser | so if you boot a machine, local storage gets /dev/vda, configdrive is automatically setup (no manual attach) as /dev/vdb .. once volume-attach is called, it is "appended" to the config so the volume appears at /dev/vdc | 04:07 |
| mnaser | when rebooting, it creates the libvirt config with /dev/vda for local, /dev/vdb for attached storage, and /dev/vdc (last one) for configdrive | 04:08 |
| thingee | huh weird | 04:08 |
| fungi | or at least that last step is what we want, however for some reason even without reboot it's attaching the new cinder volume at /dev/vdb even though the configdrive is already using /dev/vdb | 04:08 |
| *** thorst_ has joined #openstack-infra | 04:08 | |
| *** sripriya_ has joined #openstack-infra | 04:09 | |
| fungi | last step of the first description i mean | 04:09 |
| fungi | afaik this is all happening without a reboot, correct clarkb? | 04:09 |
| mnaser | when using it in nova boot, i suspect this will work around it because when nova will deploy the new instance, it'll attach the volume at /dev/vdb and toss configdrive at the last position /dev/vdc | 04:10 |
| *** sripriya has quit IRC | 04:10 | |
| fungi | so nova is indicating it attached the cinder volume at /dev/vdb even though it was already using /dev/vdb for the configdrive | 04:10 |
| calebb | fungi: clarkb says "Yes no reboot, I can't computer, too much hamburger" | 04:10 |
| fungi | mmmhamburger | 04:10 |
| fungi | also hey calebb! | 04:10 |
| calebb | hi | 04:10 |
| mnaser | fungi: yes, it's a bit complicated because what happens is that when the disk gets attached, openstack tells libvirt to attach a drive, the newly attached drive will appear as /dev/vdc in the OS (because /dev/vdb already is used by configdrive) | 04:11 |
| mnaser | we've ran into this a few times with customers, its quite an annoyance | 04:11 |
| *** annegentl_ has quit IRC | 04:11 | |
| *** sflanigan has joined #openstack-infra | 04:11 | |
| mnaser | especially when they update their fstab files, reboot and are greeted with a machine that refuses to continue to boot because /dev/vdc is no longer the device they thought it once was | 04:11 |
| fungi | yeah, essentially nova is confused about the device name the volume actually ended up at in the guest | 04:12 |
| mnaser | if you ask me, nova should never be involved in device names | 04:13 |
| mnaser | because so many difference operating systems can deal about this in many different ways | 04:13 |
| fungi | it does seem overreaching to me too, yes | 04:13 |
| mnaser | /dev/vdb makes no sense for a windows server | 04:13 |
| fungi | if anything, it should keep to biosdevnames | 04:14 |
| fungi | since it really has no visibility above the hypervisor level | 04:14 |
| *** gokrokve has joined #openstack-infra | 04:15 | |
| mnaser | yep. hell libvirt doesnt even know its at /dev/vdb | 04:15 |
| mnaser | looking at the libvirt config for the config drive.. i notice this | 04:15 |
| mnaser | <target dev='vdz' bus='virtio'/> | 04:15 |
| mnaser | yet in the os... it's located on /dev/vdc not /dev/vdz | 04:15 |
| *** thorst_ has quit IRC | 04:16 | |
| *** annegentl_ has joined #openstack-infra | 04:17 | |
| *** aeng has joined #openstack-infra | 04:17 | |
| clarkb | ok I couldn't not try this real quick and added a step to script to install lvm2 | 04:18 |
| clarkb | if this works I can sleep a happy person | 04:18 |
| clarkb | if it breaks then tomorrow morning I have my mission | 04:18 |
| mnaser | cool clarkb let me know | 04:19 |
| mnaser | https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1035364 | 04:19 |
| openstack | Launchpad bug 1035364 in libvirt (Ubuntu) "guest display wrong disk name than expected" [High,Invalid] | 04:19 |
| mnaser | this kinda sums it up | 04:19 |
| clarkb | woot! puppet is running and doing stuff | 04:25 |
| *** ianw has quit IRC | 04:25 | |
| clarkb | the reboot test on device order will be important | 04:25 |
| mnaser | the order should be ok post-reboot by my experience but we shall see! | 04:26 |
| openstackgerrit | Clark Boylan proposed openstack-infra/shade: Invalidate volume cache when waiting for attach https://review.openstack.org/286977 | 04:27 |
| clarkb | mordred: ^ thas one of the patches | 04:27 |
| *** ianw has joined #openstack-infra | 04:28 | |
| *** Jeffrey4l has joined #openstack-infra | 04:28 | |
| ianw | testing | 04:30 |
| clarkb | ianw: testing fix for swift uploads thing? | 04:30 |
| ianw | clarkb: sorry, no, testing that my linode irc bouncer is back | 04:30 |
| clarkb | ah | 04:30 |
| clarkb | test is successful then | 04:31 |
| ianw | clarkb: but i did test with that change, and it worked | 04:31 |
| ianw | but i haven't figured out how to really expire a token | 04:31 |
| ianw | (yet) | 04:31 |
| *** chenli has joined #openstack-infra | 04:32 | |
| clarkb | http://162.253.54.85/ /me does a happy dance | 04:33 |
| *** cody-somerville has quit IRC | 04:33 | |
| clarkb | adding to dns now, this means tomorrow we can nodepool this cloud \o/ | 04:33 |
| mnaser | woo | 04:34 |
| *** annegentl_ has quit IRC | 04:34 | |
| clarkb | http://mirror.ca-ymq-1.vexxhost.openstack.org will be a thing as soon as dns update propogates | 04:34 |
| *** cody-somerville has joined #openstack-infra | 04:35 | |
| mnaser | clarkb: cool, we also have a local ubuntu mirror -- http://ubuntu.mirror.vexxhost.com/ -- if you want to use that instead of maintaining one | 04:36 |
| *** annegentl_ has joined #openstack-infra | 04:36 | |
| clarkb | mnaser: we already maintain one that we expose out via afs everywhere | 04:36 |
| mnaser | oh i see | 04:36 |
| clarkb | mnaser: we need to do it for various other cloud regions so its a thing | 04:36 |
| mnaser | http://mirror.ca-ymq-1.vexxhost.openstack.org/ is loading for me so that's sweet | 04:36 |
| clarkb | mnaser: the big thing in that mirror is the whele cache though | 04:36 |
| fungi | at this point the ubuntu mirror is baked into the same afs volumes we use for pypi mirroring, pre-built wheel mirroring, soon rh/epel and npm | 04:37 |
| fungi | easier if we just drop in one that serves it all | 04:37 |
| mnaser | i see, anyways, on that note, i have to head off to bed, long day tomorrow deploying a bunch of clouds | 04:37 |
| fungi | thanks again and have a great night! | 04:38 |
| mnaser | night folks, i should be around on irc tomorrow if anything | 04:38 |
| mnaser | no problem | 04:38 |
| *** oomichi_ has joined #openstack-infra | 04:40 | |
| *** yamahata has joined #openstack-infra | 04:41 | |
| *** gokrokve_ has joined #openstack-infra | 04:44 | |
| *** gokrokve has quit IRC | 04:47 | |
| *** tiswanso has quit IRC | 04:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add support to shade-launch-node for cinder attach https://review.openstack.org/285477 | 04:50 |
| clarkb | and thats the rest of my patches to make this work | 04:50 |
| *** annegentl_ has quit IRC | 04:50 | |
| *** links has joined #openstack-infra | 04:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Also reset swift service object at upload time https://review.openstack.org/286964 | 04:51 |
| clarkb | and now for the add to nodepool patch | 04:51 |
| *** camunoz has quit IRC | 04:52 | |
| *** rossella_s has quit IRC | 04:53 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip long-running jobs on neutronclient doc changes https://review.openstack.org/286982 | 04:54 |
| *** rossella_s has joined #openstack-infra | 04:54 | |
| *** Jeffrey4l has quit IRC | 04:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Also reset swift service object at upload time https://review.openstack.org/286964 | 04:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add vexxhost to nodepool with max servers 1 https://review.openstack.org/286983 | 04:56 |
| *** dchen has quit IRC | 04:56 | |
| clarkb | if we get ^ in tonight it will be the fastest ne wcloud ever | 04:56 |
| clarkb | but I cna't pay attention to it so probably not a great idea | 04:56 |
| *** Sukhdev has joined #openstack-infra | 04:57 | |
| *** fawadkhaliq has quit IRC | 04:58 | |
| *** pahuang has quit IRC | 04:58 | |
| fungi | i'm around and will keep an eye on it | 04:59 |
| fungi | so approved | 04:59 |
| fungi | mordred: ^ you may also be awake | 04:59 |
| fungi | and i assume jhesketh is awake since this is, like, lunchtime for him or some such | 05:01 |
| *** ashleighfarnham has quit IRC | 05:01 | |
| *** dimtruck is now known as zz_dimtruck | 05:02 | |
| clarkb | first thing it will do is upload images | 05:03 |
| *** kzaitsev_mb has joined #openstack-infra | 05:03 | |
| clarkb | then it should try booting things which should just work because vexxhost uses 'public' which is the magic string nodepool unerstands for IPs | 05:04 |
| fungi | or shade understands | 05:04 |
| fungi | i never can keep straight where that logic extraction line was drawn | 05:04 |
| *** kzaitsev_mb has quit IRC | 05:08 | |
| *** sdake has quit IRC | 05:09 | |
| jhesketh | fungi: yep, I'm around, happy to keep an eye on it | 05:09 |
| jhesketh | clarkb: ^ | 05:09 |
| clarkb | jhesketh: cool, I expect worst case that it just doesn't work for some reason which will prevent jobs on it and should be fine | 05:09 |
| clarkb | it should fail safe | 05:09 |
| *** pahuang has joined #openstack-infra | 05:10 | |
| *** dchen has joined #openstack-infra | 05:10 | |
| jhesketh | clarkb: yeah, or nodepool gets stuck on a bad cloud like over the weekend | 05:12 |
| jhesketh | but that's easy to rollback | 05:12 |
| *** thorst_ has joined #openstack-infra | 05:14 | |
| *** dchen has quit IRC | 05:15 | |
| *** amotoki has joined #openstack-infra | 05:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Add vexxhost to nodepool with max servers 1 https://review.openstack.org/286983 | 05:17 |
| *** salv-orl_ has joined #openstack-infra | 05:19 | |
| clarkb | well thats exciting, looks like openstack ansible inventory may be broken? | 05:19 |
| clarkb | and I broke it \o/ | 05:19 |
| clarkb | Inventory script (/etc/ansible/hosts/openstack) had an execution error: Error fetching server list on openstackci-osic:RegionOne: | 05:20 |
| clarkb | https://review.openstack.org/#/c/286881/1 is the fix I think | 05:21 |
| *** thorst_ has quit IRC | 05:21 | |
| clarkb | rechecked it as the one job there timed out | 05:21 |
| fungi | osic keystone endpoint craziness? | 05:21 |
| clarkb | fungi: I got project_name and project_id mixed up :/ | 05:22 |
| *** salv-orlando has quit IRC | 05:22 | |
| fungi | oh | 05:22 |
| clarkb | https://review.openstack.org/#/c/286881/1 sorts it out | 05:22 |
| fungi | that's easy to do, yeah | 05:22 |
| clarkb | looking at the timing though I don't understand why it started when it did | 05:25 |
| fungi | cache refresh? | 05:25 |
| clarkb | oh maybe ya | 05:26 |
| *** piet has joined #openstack-infra | 05:26 | |
| clarkb | oh and that won't apply without manual intervention | 05:28 |
| clarkb | fun | 05:28 |
| clarkb | I will edit ansible-clouds.yaml by hand now | 05:28 |
| fungi | and possibly also delete the cache (or i guess there is no cache update happening yet) | 05:29 |
| *** harlowja_at_home has joined #openstack-infra | 05:29 | |
| clarkb | I see why it broke when it did. I updated hiera to point at new key but puppet sitll uses old key which results in empty string | 05:29 |
| clarkb | jhesketh: if you can review https://review.openstack.org/#/c/286881/1 that would be plenty helpful in correcting the puppetmaster weirdness | 05:31 |
| jhesketh | clarkb: looking | 05:33 |
| jhesketh | clarkb: +w | 05:34 |
| clarkb | jhesketh: thank you | 05:34 |
| *** sridhar_ram1 has joined #openstack-infra | 05:34 | |
| jhesketh | clarkb: you might need to edit it by hand to get it to apply, but you seem to have done that already | 05:34 |
| clarkb | yup | 05:35 |
| clarkb | prepared to do it again too | 05:35 |
| *** achanda has quit IRC | 05:35 | |
| clarkb | I did confirm that my manual edit got puppet running on the last cron fire | 05:35 |
| clarkb | but I think it will rebreak things because that patch hasnt merged yet | 05:35 |
| *** pcaruana has joined #openstack-infra | 05:36 | |
| jhesketh | clarkb: the inventory is cached for somethign like 24 hours, so you probably won't need to do it again unless you need to pick up more changes | 05:36 |
| jhesketh | ie the clouds files will update, but the ansible inventory won't look or care about it | 05:36 |
| clarkb | ah right | 05:36 |
| jhesketh | we should possible get puppet to clear that cache on changes to the cloud files | 05:37 |
| *** ayoung has quit IRC | 05:38 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix spurios = in dib-python readme https://review.openstack.org/286991 | 05:38 |
| *** abregman has joined #openstack-infra | 05:38 | |
| *** achanda has joined #openstack-infra | 05:39 | |
| *** achanda has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack-infra/system-config: Use project_name not _id with OSIC https://review.openstack.org/286881 | 05:41 |
| *** abregman has quit IRC | 05:42 | |
| *** piet has quit IRC | 05:42 | |
| clarkb | oh right the nodepool buidler does not reload its config | 05:43 |
| * clarkb will restart it | 05:43 | |
| *** abregman has joined #openstack-infra | 05:44 | |
| clarkb | and with that uplaods are uploading | 05:44 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Reorder developer quickstart docs https://review.openstack.org/286997 | 05:45 |
| clarkb | I should work on fixing that | 05:45 |
| greghaynes | clarkb: oops | 05:45 |
| clarkb | greghaynes: basically have to restart service whenever we change cloud prvider details | 05:45 |
| greghaynes | clarkb: it should be easy enough if we can set a timeout on waiting for a gear job | 05:45 |
| clarkb | ya | 05:46 |
| *** jcooley_ has joined #openstack-infra | 05:47 | |
| *** jcooley_ has left #openstack-infra | 05:47 | |
| *** jcooley_ has joined #openstack-infra | 05:47 | |
| *** vikram_ has joined #openstack-infra | 05:47 | |
| vikram_ | I need some real openstack deployment examples.. Anyone can help? | 05:48 |
| clarkb | vikram_: for deploying clouds? devstsack, the openstack puppet modules, openstack ansible modules, chef modules ,etc should all provide examples | 05:48 |
| *** pcaruana has quit IRC | 05:49 | |
| vikram_ | clarkb: We need to deploy for one of our customers | 05:49 |
| vikram_ | clarkb: Need to know what's industry best practice | 05:49 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix cloud-init-disable-resizefs README title https://review.openstack.org/286999 | 05:49 |
| *** gokrokve has joined #openstack-infra | 05:50 | |
| clarkb | vikram_: I think that largely depends on your specific needs. There are definitely some things that peopel seem to chose consistently though. KVM, rabbitmq, mysql for example | 05:50 |
| vikram_ | clarkb: Precisely, what & where services need to run to have better perf and scalability | 05:50 |
| vikram_ | clarkb: Any live deployment scenario I can find somewhere for reference | 05:51 |
| *** rcernin has joined #openstack-infra | 05:51 | |
| clarkb | vikram_: http://docs.openstack.org/arch-design/ may help | 05:52 |
| openstackgerrit | Merged openstack/requirements: Bump up glance_store to 0.12.0 https://review.openstack.org/286272 | 05:52 |
| *** gokrokve_ has quit IRC | 05:53 | |
| vikram_ | clarkb: Ok.. Will have a look .. Thanks! | 05:53 |
| *** gokrokve_ has joined #openstack-infra | 05:53 | |
| openstackgerrit | Merged openstack/requirements: bump aodhclient upper-constraints https://review.openstack.org/286263 | 05:53 |
| *** zz_dimtruck is now known as dimtruck | 05:53 | |
| *** esikachev has joined #openstack-infra | 05:54 | |
| *** sarob has joined #openstack-infra | 05:54 | |
| clarkb | OpenStackCloudException: Image creation failed: Error finding address for https://image-ca-ymq-1.vexxhost.net/v2/images/1b1e8768-df21-4252-87c2-89ee76d2a7a8/file: Unable to establish connection to https://image-ca-ymq-1.vexxhost.net/v2/images/1b1e8768-df21-4252-87c2-89ee76d2a7a8/file | 05:55 |
| *** gokrokve has quit IRC | 05:55 | |
| clarkb | not sure if that is consistent yet | 05:55 |
| clarkb | I confirmed that locally I can establish a tcp connection to that host on port 443 | 05:55 |
| *** gokrokve has joined #openstack-infra | 05:56 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add ability to check for mount leaks https://review.openstack.org/280018 | 05:56 |
| clarkb | looks like it just happened again so we may want to consider setting max-servers to -1 while we figure that out | 05:57 |
| *** gokrokve_ has quit IRC | 05:57 | |
| *** Sukhdev has quit IRC | 05:57 | |
| *** Sukhdev has joined #openstack-infra | 05:58 | |
| *** esikachev has quit IRC | 05:58 | |
| clarkb | but I really need to not computer anymore. If we don't set max-servers to -1 we will juist attempt to upload images in a tight loop over and over assuming the fail is consistent | 05:59 |
| *** sarob has quit IRC | 05:59 | |
| clarkb | max-servers: -1 will stop uploads | 05:59 |
| *** achanda has joined #openstack-infra | 06:00 | |
| *** camunoz has joined #openstack-infra | 06:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Disable image uplaods to vexxhost https://review.openstack.org/287003 | 06:03 |
| clarkb | jhesketh: ^ if you wish to disable it | 06:03 |
| clarkb | I did manage to uplaod successfully using openstackclient too so dunno what is going on yet | 06:04 |
| jhesketh | clarkb: lets disable it for now | 06:04 |
| *** kzaitsev_mb has joined #openstack-infra | 06:04 | |
| jhesketh | clarkb: will it still try and build images even when it's disabled though? | 06:04 |
| clarkb | jhesketh: dib will build images because the other cloud sneed them, nodepool just won't upload the results to vexxhost with max-servers set to -1 | 06:05 |
| clarkb | if we set max-servers to 0 it will upload images but not boot any test slaves | 06:05 |
| *** jaosorior has joined #openstack-infra | 06:06 | |
| jhesketh | ah cool, so we don't need to remove it from the image providers | 06:06 |
| jhesketh | (or the images from the provider) | 06:06 |
| clarkb | correct | 06:06 |
| clarkb | I wonder if this is some form of rate limiting as upload for the first image in the list seems to still be going strong | 06:06 |
| *** kzaitsev_mb has quit IRC | 06:09 | |
| clarkb | jhesketh: I actually have a hunch it is a rate limiting strategy since the first upload seems to be going strong and only the others fail | 06:10 |
| jhesketh | rate limit to 1 though? | 06:10 |
| clarkb | jhesketh: we can likely confirm that with mnaser tomorrow (I would still go ahead and merge the max-servers: -1 change) | 06:10 |
| clarkb | jhesketh: ya one upload at a time seems like | 06:11 |
| jhesketh | clarkb: already +w'd it | 06:11 |
| jhesketh | that'd be a surprising value, but maybe | 06:11 |
| *** binbincong has quit IRC | 06:13 | |
| *** dimtruck is now known as zz_dimtruck | 06:17 | |
| *** thorst_ has joined #openstack-infra | 06:19 | |
| *** dingyichen has quit IRC | 06:22 | |
| *** gnuoy_ has joined #openstack-infra | 06:23 | |
| *** gongysh has quit IRC | 06:24 | |
| *** binbincong has joined #openstack-infra | 06:24 | |
| *** exploreshaifali has joined #openstack-infra | 06:25 | |
| *** gongysh has joined #openstack-infra | 06:26 | |
| *** thorst_ has quit IRC | 06:26 | |
| *** kushal has joined #openstack-infra | 06:28 | |
| *** gnuoy_ has quit IRC | 06:30 | |
| *** HeOS has joined #openstack-infra | 06:32 | |
| *** amotoki has quit IRC | 06:33 | |
| *** amotoki has joined #openstack-infra | 06:34 | |
| *** armax has quit IRC | 06:35 | |
| *** amotoki has quit IRC | 06:36 | |
| *** krtaylor has quit IRC | 06:37 | |
| *** jamielennox is now known as jamielennox|away | 06:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/287011 | 06:41 |
| *** gnuoy_ has joined #openstack-infra | 06:43 | |
| *** ybathia has joined #openstack-infra | 06:44 | |
| *** gnuoy_ has quit IRC | 06:44 | |
| *** HeOS has quit IRC | 06:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable image uplaods to vexxhost https://review.openstack.org/287003 | 06:47 |
| *** ybathia has quit IRC | 06:47 | |
| *** zeih has joined #openstack-infra | 06:47 | |
| *** jamielennox|away is now known as jamielennox | 06:49 | |
| *** krtaylor has joined #openstack-infra | 06:49 | |
| *** hichihara has joined #openstack-infra | 06:49 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/285901 | 06:53 |
| *** sridhar_ram1 is now known as sridhar_ram | 06:54 | |
| *** gokrokve has quit IRC | 06:56 | |
| *** shardy has joined #openstack-infra | 07:00 | |
| *** chenli has quit IRC | 07:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:05 | |
| jamespage | yolanda, when your around - could you take a peek at https://review.openstack.org/#/c/286853 | 07:05 |
| jamespage | beisner's CI for charms needs to be able to provide verifcation feedback... | 07:06 |
| jamespage | thanks :-) | 07:06 |
| *** scheuran has joined #openstack-infra | 07:06 | |
| *** e0ne has joined #openstack-infra | 07:06 | |
| *** jamielennox is now known as jamielennox|away | 07:08 | |
| *** kzaitsev_mb has quit IRC | 07:10 | |
| *** gildub has quit IRC | 07:10 | |
| *** nmagnezi has joined #openstack-infra | 07:12 | |
| *** amotoki has joined #openstack-infra | 07:14 | |
| *** oomichi_ has quit IRC | 07:16 | |
| yolanda | hi jamespage, going to take a look | 07:18 |
| yolanda | ah, AJaeger was faster | 07:19 |
| *** jtomasek has joined #openstack-infra | 07:19 | |
| jamespage | lol - thanks anyway yolanda! | 07:22 |
| *** thorst_ has joined #openstack-infra | 07:23 | |
| *** maishsk has joined #openstack-infra | 07:24 | |
| yolanda | next time :) | 07:26 |
| *** thorst_ has quit IRC | 07:31 | |
| *** pahuang has quit IRC | 07:33 | |
| *** amotoki has quit IRC | 07:35 | |
| *** HeOS has joined #openstack-infra | 07:36 | |
| *** [1]Thelo has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack-infra/project-config: add release notes jobs for solum https://review.openstack.org/243301 | 07:38 |
| *** Thelo has quit IRC | 07:40 | |
| *** [1]Thelo is now known as Thelo | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable verified label for charms https://review.openstack.org/286853 | 07:42 |
| *** dingyichen has joined #openstack-infra | 07:42 | |
| *** amotoki has joined #openstack-infra | 07:43 | |
| AJaeger | Sorry, yolanda ;) | 07:45 |
| AJaeger | lifeless: yes, it'S basically just change tox, see glance, cinder or nova on how it was done there. | 07:46 |
| *** mikelk has joined #openstack-infra | 07:47 | |
| *** amotoki has quit IRC | 07:48 | |
| *** esikachev has joined #openstack-infra | 07:50 | |
| *** harlowja_at_home has quit IRC | 07:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/287011 | 07:56 |
| *** kingia_ has quit IRC | 07:56 | |
| *** kingia has joined #openstack-infra | 07:56 | |
| *** markvoelker has quit IRC | 07:58 | |
| *** jcoufal has joined #openstack-infra | 07:58 | |
| *** e0ne has quit IRC | 07:59 | |
| *** pcaruana has joined #openstack-infra | 08:02 | |
| *** achanda has quit IRC | 08:03 | |
| *** hichihara has quit IRC | 08:04 | |
| lifeless | AJaeger: so I'm not sure what you're waiting on me to email ? | 08:05 |
| *** kzaitsev_mb has joined #openstack-infra | 08:06 | |
| *** exploreshaifali has quit IRC | 08:06 | |
| *** mrmartin has joined #openstack-infra | 08:07 | |
| AJaeger | lifeless: I think we should write an email to tell projects how to move forward with constraints - with just updating tox.ini. | 08:07 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 08:07 |
| AJaeger | That's what we discussed with sdague last week or so - and concluded with postponing it to first figure out whether everything works | 08:07 |
| *** achanda has joined #openstack-infra | 08:07 | |
| AJaeger | Or what do you see as best next step forward? | 08:08 |
| *** Sukhdev has quit IRC | 08:09 | |
| *** vgridnev has joined #openstack-infra | 08:09 | |
| *** sridhar_ram has quit IRC | 08:10 | |
| *** Sukhdev has joined #openstack-infra | 08:10 | |
| *** kzaitsev_mb has quit IRC | 08:11 | |
| *** zz_dimtruck is now known as dimtruck | 08:11 | |
| *** maishsk_ has joined #openstack-infra | 08:12 | |
| *** Sukhdev has quit IRC | 08:13 | |
| *** maishsk has quit IRC | 08:13 | |
| *** maishsk_ is now known as maishsk | 08:13 | |
| *** matrohon has joined #openstack-infra | 08:14 | |
| jamespage | beisner, the swift-proxy reverse-mirror to bzr should be fixed now | 08:15 |
| jamespage | its a workaround.. | 08:15 |
| *** [HeOS] has joined #openstack-infra | 08:15 | |
| lifeless | AJaeger: I was encouraging Nakato to write such a mail, but then the style changed with the discussion w/nova about target namings | 08:15 |
| *** dingyichen has quit IRC | 08:15 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs https://review.openstack.org/286136 | 08:16 |
| AJaeger | lifeless: I see. Will she follow up now that this is settled? | 08:17 |
| lifeless | AJaeger: lets ask :) | 08:18 |
| lifeless | Nakato: ^ | 08:18 |
| AJaeger | lifeless, Nakato: both of you did a great job getting us to this point, thanks! | 08:18 |
| *** HeOS has quit IRC | 08:18 | |
| *** achanda has quit IRC | 08:18 | |
| *** dingyichen has joined #openstack-infra | 08:19 | |
| *** korzen has joined #openstack-infra | 08:19 | |
| *** sarob has joined #openstack-infra | 08:21 | |
| *** esp has joined #openstack-infra | 08:25 | |
| *** jamielennox|away is now known as jamielennox | 08:25 | |
| *** vincentll has joined #openstack-infra | 08:25 | |
| *** dingyichen has quit IRC | 08:25 | |
| *** tcammann_ has joined #openstack-infra | 08:25 | |
| *** jlanoux has joined #openstack-infra | 08:26 | |
| AJaeger | lifeless: can you discuss with here later, please? Getting the three of us together at one time seems to be hard with time zones ;/ | 08:26 |
| *** sarob has quit IRC | 08:27 | |
| *** NobodyCa1 has joined #openstack-infra | 08:28 | |
| *** thorst_ has joined #openstack-infra | 08:29 | |
| *** lxsli0 has joined #openstack-infra | 08:32 | |
| *** esp has quit IRC | 08:32 | |
| *** bastelfreak has joined #openstack-infra | 08:32 | |
| *** tcammann has joined #openstack-infra | 08:32 | |
| *** NobodyCa1 has quit IRC | 08:33 | |
| *** NobodyCa1 has joined #openstack-infra | 08:34 | |
| *** arxcruz has joined #openstack-infra | 08:35 | |
| *** thorst_ has quit IRC | 08:35 | |
| *** Daisy has joined #openstack-infra | 08:36 | |
| *** kushal has quit IRC | 08:36 | |
| *** dizquierdo has joined #openstack-infra | 08:37 | |
| *** lxsli0 has quit IRC | 08:38 | |
| *** tcammann_ has quit IRC | 08:38 | |
| *** esikachev has quit IRC | 08:38 | |
| openstackgerrit | Stefan Andres proposed openstack-infra/shade: Fix lookup for network parameter in neutron_available_floating_ips https://review.openstack.org/280178 | 08:38 |
| *** tcammann has quit IRC | 08:39 | |
| *** chenli has joined #openstack-infra | 08:39 | |
| *** NobodyCa1 has quit IRC | 08:39 | |
| *** ifarkas has joined #openstack-infra | 08:40 | |
| *** flepied1 has quit IRC | 08:48 | |
| *** peterlisak has quit IRC | 08:48 | |
| *** onovy has quit IRC | 08:48 | |
| *** exploreshaifali has joined #openstack-infra | 08:48 | |
| *** fhubik has joined #openstack-infra | 08:49 | |
| *** esikachev has joined #openstack-infra | 08:50 | |
| *** hashar has joined #openstack-infra | 08:50 | |
| *** ihrachys has joined #openstack-infra | 08:51 | |
| *** rossella_s has quit IRC | 08:54 | |
| *** dimtruck is now known as zz_dimtruck | 08:54 | |
| *** yaume has joined #openstack-infra | 08:54 | |
| *** rossella_s has joined #openstack-infra | 08:54 | |
| *** markvoelker has joined #openstack-infra | 08:58 | |
| openstackgerrit | JF Taltavull proposed openstack-infra/project-config: Add irc channel for the i18n french team https://review.openstack.org/287048 | 08:59 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Adjust acls for fuel-noop-fixtures https://review.openstack.org/286109 | 09:00 |
| *** zeih has quit IRC | 09:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip long-running jobs on neutronclient doc changes https://review.openstack.org/286982 | 09:04 |
| *** gokrokve has joined #openstack-infra | 09:05 | |
| *** asettle has joined #openstack-infra | 09:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip dsvm jobs on release note only tempest changes https://review.openstack.org/286831 | 09:05 |
| openstackgerrit | Merged openstack-infra/project-config: update django_openstack_auth to use publish-to-pypi jobs https://review.openstack.org/286747 | 09:05 |
| *** lucas-dinner is now known as lucasagomes | 09:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:07 | |
| *** daemontool_ has joined #openstack-infra | 09:08 | |
| *** daemontool_ is now known as daemontool | 09:08 | |
| *** gokrokve has quit IRC | 09:09 | |
| *** derekh has joined #openstack-infra | 09:10 | |
| *** Daisy has quit IRC | 09:11 | |
| *** daemontool__ has quit IRC | 09:11 | |
| *** Daisy has joined #openstack-infra | 09:11 | |
| *** esker has quit IRC | 09:11 | |
| *** esker has joined #openstack-infra | 09:12 | |
| *** kzaitsev_mb has quit IRC | 09:12 | |
| *** arxcruz has quit IRC | 09:13 | |
| *** chenli has quit IRC | 09:14 | |
| *** chenli has joined #openstack-infra | 09:15 | |
| *** Daisy has quit IRC | 09:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Sort projects in big skip list https://review.openstack.org/286527 | 09:18 |
| *** kzaitsev_mb has joined #openstack-infra | 09:18 | |
| *** zhurong has quit IRC | 09:19 | |
| AJaeger | yolanda: could you review the change above (286527) and also 285356 as time permits, please? | 09:19 |
| *** jistr has joined #openstack-infra | 09:19 | |
| yolanda | sure | 09:19 |
| *** zhurong has joined #openstack-infra | 09:20 | |
| *** vgridnev has quit IRC | 09:20 | |
| AJaeger | thanks | 09:21 |
| *** _degorenko|afk is now known as degorenko | 09:21 | |
| *** gongysh has quit IRC | 09:21 | |
| *** BobBall has joined #openstack-infra | 09:22 | |
| *** flepied has joined #openstack-infra | 09:22 | |
| *** maishsk has quit IRC | 09:26 | |
| *** arxcruz has joined #openstack-infra | 09:27 | |
| *** dizquierdo has quit IRC | 09:28 | |
| *** e0ne has joined #openstack-infra | 09:28 | |
| *** chenli has quit IRC | 09:29 | |
| openstackgerrit | C.J. Collier proposed openstack-infra/gerritbot: Improve gerritbot error handling https://review.openstack.org/287056 | 09:29 |
| tonyb | Policyish question ... I need to test a devstack plugin on the experimental queue. does that plugin need to be on git.o.o? or can it be on github for now? | 09:30 |
| *** maishsk has joined #openstack-infra | 09:30 | |
| *** vgridnev has joined #openstack-infra | 09:30 | |
| tonyb | clearly if it's on github that adds an additional failure path but is that a problem? | 09:31 |
| *** Daisy has joined #openstack-infra | 09:31 | |
| *** [HeOS] is now known as HeOS | 09:31 | |
| *** sorantis has joined #openstack-infra | 09:32 | |
| *** x00350071 is now known as xiangxinyong | 09:33 | |
| *** thorst_ has joined #openstack-infra | 09:33 | |
| *** pblaho has joined #openstack-infra | 09:33 | |
| AJaeger | tonyb: if you use export DEVSTACK_LOCAL_CONFIG="enable_plugin networking-midonet git://git.openstack.org/openstack/networking-midonet" | 09:33 |
| AJaeger | note the comment before in jenkins/jobs/networking-midonet.yaml: | 09:33 |
| *** dtardivel has joined #openstack-infra | 09:33 | |
| AJaeger | "caches in nodepool, however make it a valid url for documentation purposes." | 09:34 |
| AJaeger | And we only cache git.openstack.org repositories... | 09:34 |
| AJaeger | So, short answer: Needs to be on git.o.o | 09:34 |
| tonyb | AJaeger: Okay thanks. | 09:34 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 09:35 |
| tonyb | I knew about the cache in the nodepool image but I didn't thknk they were that tied. | 09:35 |
| * tonyb goes and does the new project dance. | 09:35 | |
| *** e0ne has quit IRC | 09:37 | |
| *** thorst_ has quit IRC | 09:40 | |
| *** exploreshaifali has quit IRC | 09:41 | |
| *** zhurong has quit IRC | 09:44 | |
| *** kashyap has joined #openstack-infra | 09:44 | |
| *** fabio_ has joined #openstack-infra | 09:45 | |
| *** kzaitsev_mb has quit IRC | 09:45 | |
| *** lane_kong is now known as lynn | 09:45 | |
| kashyap | Heya, both the Zuul urls mentioned in the "Before You Start" section are 404 - http://docs.openstack.org/infra/manual/creators.html | 09:45 |
| *** Qiming has quit IRC | 09:45 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:45 | |
| *** lynn is now known as Guest29970 | 09:46 | |
| *** Guest29970 is now known as lynn_kong | 09:46 | |
| AJaeger | kashyap: Oops, my fault ;( Do you speak RST and want to send a patch for infra-manual? | 09:47 |
| kashyap | AJaeger: Sure, rST is just all indentation and a tiny bit of syntax :-) | 09:48 |
| *** vincentll has quit IRC | 09:48 | |
| * kashyap thought I'd begin my day with debugging a Ceph problem... | 09:48 | |
| * AJaeger screwed up that tiny bit ;) | 09:49 | |
| AJaeger | kashyap: thanks | 09:49 |
| *** sheeprine has quit IRC | 09:49 | |
| *** sheeprine has joined #openstack-infra | 09:50 | |
| *** maishsk has quit IRC | 09:53 | |
| *** maishsk has joined #openstack-infra | 09:53 | |
| *** fhubik is now known as fhubik_brb | 09:55 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove openstack-neutron-release from acceptable channels list https://review.openstack.org/278180 | 09:57 |
| *** _nadya_ has joined #openstack-infra | 09:58 | |
| *** vgridnev has quit IRC | 10:00 | |
| *** e0ne has joined #openstack-infra | 10:00 | |
| *** vgridnev has joined #openstack-infra | 10:00 | |
| markus_z | Zara: Are you around? | 10:01 |
| *** fabio_ has quit IRC | 10:01 | |
| *** electrofelix has joined #openstack-infra | 10:02 | |
| *** exploreshaifali has joined #openstack-infra | 10:02 | |
| *** maishsk has quit IRC | 10:02 | |
| *** maishsk has joined #openstack-infra | 10:05 | |
| *** cznewt has quit IRC | 10:06 | |
| *** kushal has joined #openstack-infra | 10:07 | |
| jamespage | yolanda, AJaeger - hey - can i get initial access to the charms-ci group please | 10:08 |
| yolanda | jamespage, i'll add you | 10:09 |
| *** kushal has quit IRC | 10:09 | |
| *** onovy has joined #openstack-infra | 10:10 | |
| yolanda | done | 10:10 |
| *** kushal has joined #openstack-infra | 10:11 | |
| *** lynn_kong is now known as Larrie | 10:12 | |
| *** Larrie is now known as larrie | 10:12 | |
| jamespage | yolanda, thankyou | 10:12 |
| *** peterlisak has joined #openstack-infra | 10:12 | |
| *** Jeffrey4l has joined #openstack-infra | 10:13 | |
| jamespage | beisner, uosci is now in the right group | 10:13 |
| *** kzaitsev_mb has quit IRC | 10:14 | |
| *** vincentll has joined #openstack-infra | 10:14 | |
| *** sarob has joined #openstack-infra | 10:16 | |
| *** cznewt has joined #openstack-infra | 10:17 | |
| *** amotoki has joined #openstack-infra | 10:18 | |
| *** rguillebert has joined #openstack-infra | 10:18 | |
| *** sarob has quit IRC | 10:20 | |
| *** Daisy has quit IRC | 10:20 | |
| *** fhubik_brb is now known as fhubik | 10:21 | |
| *** Daisy has joined #openstack-infra | 10:21 | |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| *** Daisy has quit IRC | 10:25 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack-infra/infra-manual: creators.rst: Fix broken internel hyperlinks https://review.openstack.org/287083 | 10:25 |
| kashyap | AJaeger: ^ Hope I got the syntax correctly this time :-) | 10:26 |
| *** larrie is now known as larrie_kong | 10:28 | |
| *** vgridnev has quit IRC | 10:31 | |
| *** vgridnev has joined #openstack-infra | 10:32 | |
| *** jordanP has joined #openstack-infra | 10:32 | |
| lucasagomes | hi all, there's a devstack-gate which will allow ironic to start testing it's ipmitool drivers in gate (it's only 3LOC) if you have a time please take a look https://review.openstack.org/#/c/284036/ thank you | 10:34 |
| korzen | hi anteaya, can you help merging Neutron Grenade multinode DVR experimental job? https://review.openstack.org/#/c/250215 | 10:36 |
| *** thorst_ has joined #openstack-infra | 10:38 | |
| Zara | markus_z: I am now! | 10:42 |
| markus_z | Zara: I'm going to lunch in a few minutes, are you free in 45 minutes? We could shortly discuss the requirements if you want. | 10:43 |
| *** esikache1 has joined #openstack-infra | 10:43 | |
| *** esikachev has quit IRC | 10:43 | |
| *** _nadya__ has joined #openstack-infra | 10:43 | |
| *** _nadya_ has quit IRC | 10:43 | |
| dguitarbite | AJaeger: How much more time should it take for this patch to get merged? https://review.openstack.org/#/c/281721/ | 10:44 |
| Zara | markus_z: ah, cool, works for me! | 10:45 |
| markus_z | Zara: great, see you then | 10:45 |
| *** thorst_ has quit IRC | 10:45 | |
| *** ihrachys has quit IRC | 10:47 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Move Manila experimental jobs to check pipeline https://review.openstack.org/287099 | 10:49 |
| *** Qiming has joined #openstack-infra | 10:49 | |
| Thelo | HI all, in nodepool each time I try to build an image ( via command line ) I get the following logs, and then it stucks ( http://paste.openstack.org/show/488926/ ) does anyone know why? | 10:49 |
| *** d0ugal has quit IRC | 10:50 | |
| *** d0ugal has joined #openstack-infra | 10:50 | |
| *** d0ugal has quit IRC | 10:50 | |
| *** d0ugal has joined #openstack-infra | 10:50 | |
| *** yamahata has quit IRC | 10:51 | |
| *** dims_ has joined #openstack-infra | 10:51 | |
| *** dizquierdo has joined #openstack-infra | 10:52 | |
| *** hichihara has joined #openstack-infra | 10:54 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/project-config: Publish renderspec docs https://review.openstack.org/287105 | 10:54 |
| *** |-paul-| has joined #openstack-infra | 10:55 | |
| *** fhubik is now known as fhubik_brb | 10:57 | |
| *** binbincong has quit IRC | 10:57 | |
| *** esikache1 has quit IRC | 11:00 | |
| *** sdague has joined #openstack-infra | 11:02 | |
| *** esikache1 has joined #openstack-infra | 11:04 | |
| *** fhubik_brb is now known as fhubik | 11:05 | |
| *** sorantis has quit IRC | 11:05 | |
| *** gildub has joined #openstack-infra | 11:07 | |
| *** Daisy has joined #openstack-infra | 11:08 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add gate job for Cinder Functional tests https://review.openstack.org/287115 | 11:09 |
| *** _amrith_ is now known as amrith | 11:10 | |
| *** _nadya__ has quit IRC | 11:10 | |
| *** binbincong has joined #openstack-infra | 11:10 | |
| *** daemontool_ has joined #openstack-infra | 11:15 | |
| *** sorantis has joined #openstack-infra | 11:17 | |
| *** daemontool has quit IRC | 11:17 | |
| openstackgerrit | David Caro proposed openstack-infra/jenkins-job-builder: Adding the build-discarder property https://review.openstack.org/273952 | 11:18 |
| AJaeger | dguitarbite: I hope I remember discussing later today - feel free to do this yourself as well. | 11:19 |
| *** hashar has quit IRC | 11:19 | |
| *** salv-orlando has joined #openstack-infra | 11:19 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:19 | |
| *** Daisy has quit IRC | 11:21 | |
| *** Daisy has joined #openstack-infra | 11:21 | |
| *** salv-orl_ has quit IRC | 11:22 | |
| *** Daisy has quit IRC | 11:26 | |
| *** sambetts|afk is now known as sambetts | 11:27 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** vikram_ has quit IRC | 11:37 | |
| markus_z | Zara: ready for takeoff? | 11:38 |
| Zara | markus_z: sure :) | 11:38 |
| *** mkj_wk_ is now known as mkj | 11:38 | |
| *** daemontool_ has quit IRC | 11:39 | |
| Zara | I've avoided commenting on the etherpad directly in case that was distracting, but we can do that if useful | 11:39 |
| markus_z | cool cool, so, you wanna ask questions or should I just go through the list and explain it? | 11:39 |
| *** daemontool_ has joined #openstack-infra | 11:39 | |
| markus_z | Zara: Let's just add it directly to the etherpad | 11:39 |
| Zara | okay :) | 11:39 |
| *** ociuhandu has joined #openstack-infra | 11:39 | |
| *** _nadya_ has joined #openstack-infra | 11:40 | |
| *** fhubik is now known as fhubik_brb | 11:41 | |
| *** yamamoto has quit IRC | 11:41 | |
| *** kzaitsev_mb has quit IRC | 11:42 | |
| Zara | hm, I'm not sure whether our syntax hilighting is sufficient. you can put a markdown code block in, but I'm not sure how sophisticated it is | 11:42 |
| markus_z | Zara: Do you have a storyboard item which shows all its glory (comments, ...)? | 11:42 |
| *** cznewt has quit IRC | 11:43 | |
| *** thorst_ has joined #openstack-infra | 11:43 | |
| *** zul has quit IRC | 11:44 | |
| *** resker has joined #openstack-infra | 11:44 | |
| Zara | markus_z: this one might be useful: https://storyboard.openstack.org/#!/story/2000402 | 11:45 |
| *** _nadya_ has quit IRC | 11:45 | |
| *** zul has joined #openstack-infra | 11:45 | |
| markus_z | Zara: Events and Comments go in the same "queue"? | 11:46 |
| *** esker has quit IRC | 11:47 | |
| *** gildub has quit IRC | 11:47 | |
| Zara | markus_z: yeah, I think the idea was to keep all the history together, but it's messy. we definitely want to be able to better filter what the user sees (atm they have to change their global events preferences to filter by just comments, which is annoying) | 11:49 |
| *** cznewt has joined #openstack-infra | 11:50 | |
| Zara | also, for action history, you can see the list of timeline events, but we don't have a way to go 'show story x as it was on y date' | 11:50 |
| *** thorst_ has quit IRC | 11:51 | |
| * SotK does some vague testing and finds that storyboard seems to wrap code blocks to fit on the screen, but there is no hard limit | 11:51 | |
| Zara | so that's why I've gone 'I *think* we have this', because people can mean different things by 'history' :) | 11:51 |
| SotK | I'm happy to make it not break and enforce horizontal scrolling, if that is preferred | 11:51 |
| *** ihrachys has joined #openstack-infra | 11:52 | |
| markus_z | Zara: The mix of human comments and emitted events is confusing to me. A discussion in this story is hard this way. | 11:54 |
| *** _nadya_ has joined #openstack-infra | 11:55 | |
| AJaeger | sdague, could you review two project-config changes of mine later, please? 286527 and 285356 | 11:56 |
| *** doug-fish has joined #openstack-infra | 11:56 | |
| Zara | yeah, I want a user to be able to click 'show only comments' (and maybe default to that, while storing the whole list of events somewhere). I think the idea was to put events in the list so the history was clear (eg: if the story description updated between comments, people could be referring to different things) | 11:58 |
| sdague | AJaeger: can you provide full urls, it's faster to open from my irc client? | 11:59 |
| Zara | we have plans to go back to timeline events and improve them, though right now I don't know exactly when that will be. I suspect early in the Newton cycle, though that might be pessimistic of me. | 12:00 |
| markus_z | Zara: I really like the boards and worklists | 12:00 |
| Zara | \o/ SotK^ | 12:00 |
| SotK | :D | 12:01 |
| Zara | (I'm also planning to look at the *layout* of timeline events, hopefully next week, but there won't be any substantial changes. may make things easier to follow, though) | 12:02 |
| *** kzaitsev_mb has joined #openstack-infra | 12:02 | |
| Zara | but I make no promises! :D | 12:02 |
| Zara | anyway, thank you, that's good to hear :) | 12:02 |
| markus_z | Zara: No hurries. | 12:03 |
| *** sdake has joined #openstack-infra | 12:03 | |
| Zara | :) it's useful to see what a big project cares about; we haven't paid much attention to comments (recently, anyway) because we're small, but I can see the need for a project the size of Nova. | 12:04 |
| markus_z | Zara: As soon as a discussion (forum-like) is possible within a story, I would like to use sb instead of lp. But that's the view from a developer, I'm not involved in release mgmt. and things. | 12:04 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI https://review.openstack.org/286793 | 12:04 |
| *** gokrokve has joined #openstack-infra | 12:04 | |
| markus_z | Zara: The tasks can be set to multiple different projects, right? We have sometimes bugs which affect multiple projects at once. | 12:05 |
| *** aysyd has joined #openstack-infra | 12:05 | |
| Zara | markus_z: understood. :) and yes, tasks can be set to multiple projects. | 12:06 |
| markus_z | I like the response times. very fast | 12:07 |
| openstackgerrit | Merged openstack-infra/project-config: Adjust acls for fuel-noop-fixtures https://review.openstack.org/286109 | 12:08 |
| markus_z | Zara: I haven't checked the REST API sb provides, so no feedback from me there | 12:08 |
| *** gokrokve has quit IRC | 12:09 | |
| markus_z | Zara: well, cool, do you have any open questions for me? | 12:09 |
| Zara | markus_z: I think SotK knows more about that than I do, tbh, so he might have something to add on that. (or might not!) | 12:10 |
| Zara | er, the REST API that is | 12:10 |
| * SotK goes to write things in the API section of the etherpad | 12:11 | |
| Zara | \o/ | 12:11 |
| Zara | markus_z: I don't think so, but I'm pretty sleepy atm and might think of some later | 12:11 |
| *** sdake has quit IRC | 12:13 | |
| *** kashyap has left #openstack-infra | 12:15 | |
| *** amrith is now known as _amrith_ | 12:16 | |
| markus_z | Zara: SotK: ok, cool, maybe we can talk at the summit, see you | 12:17 |
| SotK | markus_z: thanks for the comments :) | 12:17 |
| Zara | cool, thanks again, those requirements are really helpful :) | 12:17 |
| AJaeger | sdague: https://review.openstack.org/286527 and https://review.openstack.org/285356 are full URLs | 12:21 |
| *** asettle has quit IRC | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:24 | |
| *** yamamoto_ has joined #openstack-infra | 12:25 | |
| *** erlon has joined #openstack-infra | 12:25 | |
| *** fhubik_brb is now known as fhubik | 12:26 | |
| *** Liuqing has joined #openstack-infra | 12:27 | |
| openstackgerrit | Vincent proposed openstack-infra/project-config: add Synergy project https://review.openstack.org/283157 | 12:28 |
| *** yamamoto has quit IRC | 12:29 | |
| *** kiran-r has joined #openstack-infra | 12:29 | |
| *** |-paul-| has quit IRC | 12:36 | |
| dguitarbite | AJaeger: Ok, Ill try to bring it up :) | 12:37 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Remove check requirements from Kingbird" https://review.openstack.org/285356 | 12:37 |
| *** jaosorior has quit IRC | 12:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Sort projects in big skip list https://review.openstack.org/286527 | 12:37 |
| *** jaosorior has joined #openstack-infra | 12:38 | |
| *** asettle has joined #openstack-infra | 12:39 | |
| *** hichihara has quit IRC | 12:39 | |
| *** jaosorior has quit IRC | 12:39 | |
| *** jaosorior has joined #openstack-infra | 12:40 | |
| *** claudiub has joined #openstack-infra | 12:40 | |
| *** esker has joined #openstack-infra | 12:42 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 12:42 |
| *** Daisy has joined #openstack-infra | 12:43 | |
| *** yamamoto_ has quit IRC | 12:43 | |
| *** resker has quit IRC | 12:44 | |
| *** rfolco has joined #openstack-infra | 12:44 | |
| esikache1 | hey, folks! can you merge? https://review.openstack.org/#/c/284693/ | 12:44 |
| *** edmondsw has joined #openstack-infra | 12:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Move Manila experimental jobs to check pipeline https://review.openstack.org/287099 | 12:47 |
| *** thorst_ has joined #openstack-infra | 12:48 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** markvoelker has joined #openstack-infra | 12:49 | |
| *** lucasagomes is now known as lucas-hungry | 12:51 | |
| lucas-hungry | hi all, there's a devstack-gate which will allow ironic to start testing it's ipmitool drivers in gate (it's only 3LOC) if you have a time please take a look https://review.openstack.org/#/c/284036/ thank you | 12:52 |
| *** fhubik is now known as fhubik_brb | 12:52 | |
| *** rossella_s has quit IRC | 12:53 | |
| *** rossella_s has joined #openstack-infra | 12:54 | |
| *** fawadkhaliq has joined #openstack-infra | 12:54 | |
| AJaeger | sdague: could you review lucas-hungry's change, please ? ^ | 12:54 |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| *** baoli has joined #openstack-infra | 12:56 | |
| *** Qiming has quit IRC | 12:59 | |
| *** links has quit IRC | 12:59 | |
| *** dprince has joined #openstack-infra | 13:00 | |
| *** _nadya_ has quit IRC | 13:01 | |
| sdague | lucas-hungry: is there a thing above it that demonstrates it working | 13:03 |
| *** yaume_ has joined #openstack-infra | 13:03 | |
| *** yaume has quit IRC | 13:04 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 13:05 |
| *** rhallisey has joined #openstack-infra | 13:06 | |
| *** hashar has joined #openstack-infra | 13:07 | |
| *** Daisy has quit IRC | 13:08 | |
| *** Daisy has joined #openstack-infra | 13:08 | |
| *** Jeffrey4l has quit IRC | 13:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add mirror server https://review.openstack.org/238414 | 13:12 |
| *** baoli has quit IRC | 13:12 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Used the mirror server for the centos image and git repos https://review.openstack.org/285257 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use instack-qcow2 from cache if available and appropriate https://review.openstack.org/285259 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs https://review.openstack.org/285258 | 13:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job https://review.openstack.org/229789 | 13:12 |
| *** yaume_ has quit IRC | 13:12 | |
| *** Daisy has quit IRC | 13:13 | |
| *** yaume has joined #openstack-infra | 13:13 | |
| *** Jeffrey4l has joined #openstack-infra | 13:15 | |
| *** dkranz has joined #openstack-infra | 13:15 | |
| *** baoli has joined #openstack-infra | 13:19 | |
| *** fawadkhaliq has quit IRC | 13:21 | |
| *** fhubik_brb is now known as fhubik | 13:22 | |
| *** vgridnev has quit IRC | 13:23 | |
| *** vgridnev has joined #openstack-infra | 13:23 | |
| *** kgiusti has joined #openstack-infra | 13:24 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Make gate-grenade-dsvm-neutron-multinode voting https://review.openstack.org/287184 | 13:24 |
| *** baoli has quit IRC | 13:25 | |
| *** yaume has quit IRC | 13:27 | |
| *** baoli has joined #openstack-infra | 13:27 | |
| *** yaume has joined #openstack-infra | 13:27 | |
| *** yamamoto has quit IRC | 13:28 | |
| *** baoli has quit IRC | 13:29 | |
| *** zhurong has joined #openstack-infra | 13:30 | |
| *** annegentl_ has joined #openstack-infra | 13:30 | |
| *** pradk has quit IRC | 13:32 | |
| *** pleia2 has quit IRC | 13:33 | |
| *** yaume has quit IRC | 13:33 | |
| *** yaume has joined #openstack-infra | 13:34 | |
| openstackgerrit | Oleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-tuning-box' https://review.openstack.org/286137 | 13:35 |
| *** vgridnev has quit IRC | 13:35 | |
| *** baoli has joined #openstack-infra | 13:37 | |
| *** vivekd has joined #openstack-infra | 13:37 | |
| *** piet has joined #openstack-infra | 13:38 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** yaume has quit IRC | 13:40 | |
| *** yaume has joined #openstack-infra | 13:41 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/jenkins-job-builder: Complete support of Yaml Axis Plugin https://review.openstack.org/286690 | 13:41 |
| *** esker has quit IRC | 13:43 | |
| *** esker has joined #openstack-infra | 13:44 | |
| openstackgerrit | Merged openstack-infra/shade: Add ability to pass just filename to create_image https://review.openstack.org/282907 | 13:44 |
| *** jed56 has joined #openstack-infra | 13:46 | |
| *** vgridnev has joined #openstack-infra | 13:48 | |
| *** yaume has quit IRC | 13:50 | |
| *** yaume has joined #openstack-infra | 13:51 | |
| *** _nadya_ has joined #openstack-infra | 13:51 | |
| mordred | Shrews: yah. I agree on getting releasing shade | 13:51 |
| Shrews | mordred: if we're finally doing that, there are some fixes i have up that should go in | 13:52 |
| Shrews | and some others from infra folks we should review | 13:53 |
| *** yamamoto has joined #openstack-infra | 13:54 | |
| *** rlandy has joined #openstack-infra | 13:54 | |
| *** doug-fish has quit IRC | 13:55 | |
| *** yaume has quit IRC | 13:55 | |
| *** yaume has joined #openstack-infra | 13:56 | |
| *** amitgandhinz has joined #openstack-infra | 13:57 | |
| * mordred goes lookie | 13:57 | |
| Shrews | i just approved clarkb's cache fix | 13:57 |
| *** dizquierdo has quit IRC | 13:58 | |
| Shrews | we should check for that cache bug in other areas :( | 13:58 |
| *** daemontool_ is now known as daemontool | 13:59 | |
| mordred | Shrews: ++ | 13:59 |
| mordred | Shrews: also, https://review.openstack.org/#/c/286964 looks like one we should land | 14:00 |
| *** yaume has quit IRC | 14:00 | |
| mordred | Shrews: and makes me think we shoudl also pass username/password/etc down to SwiftService like we do for swift connection now | 14:00 |
| mordred | lemme check their source nad see if that works | 14:00 |
| dmsimard | Any chance I could get a final +2+A on https://review.openstack.org/#/c/286579/ ? It's to wrap up the packstack integration jobs. We can make them voting shortly afterwards. | 14:00 |
| *** yaume has joined #openstack-infra | 14:01 | |
| *** sorantis_ has joined #openstack-infra | 14:01 | |
| *** yamamoto has quit IRC | 14:02 | |
| *** lucas-hungry is now known as lucasagomes | 14:03 | |
| *** sorantis has quit IRC | 14:04 | |
| *** sorantis_ is now known as sorantis | 14:04 | |
| lucasagomes | sdague, hi there, sorry I was having lunch... Anything that demonstrates ipmitool drivers running with devstack? | 14:04 |
| *** bapalm has quit IRC | 14:05 | |
| *** yaume has quit IRC | 14:05 | |
| lucasagomes | hmm so, the support is there, the problem is that the devstack-gate script is hardcoding the drivers (pxe_ssh or agent_ssh) | 14:05 |
| sdague | lucasagomes: right, something that depends-on the d-g change to show it all running | 14:05 |
| *** yaume has joined #openstack-infra | 14:06 | |
| sdague | because it's not something in the set of tests | 14:06 |
| *** jbernard_ has joined #openstack-infra | 14:06 | |
| *** bapalm has joined #openstack-infra | 14:06 | |
| lucasagomes | sdague, the dependency was a devstack patch ( https://review.openstack.org/#/c/280267/ ) which is already merged | 14:06 |
| lucasagomes | sdague, oh gotcha what you mean | 14:07 |
| *** weshay has joined #openstack-infra | 14:07 | |
| lucasagomes | lemme see think how I can do that... | 14:07 |
| *** yamamoto has joined #openstack-infra | 14:07 | |
| lucasagomes | sdague, we have the jobs already enabled in ironic as non-voting, if I submit a patch for ironic depending on the d-g changes you think it will get pulled? | 14:08 |
| * lucasagomes tries | 14:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add nodepool vexxhost dashboard for grafana https://review.openstack.org/287211 | 14:08 |
| sdague | lucasagomes: yeh, it should | 14:08 |
| sdague | devstack-gate changes can be tested that way | 14:09 |
| pabelanger | clarkb: fungi: jeblair: grafana dashboard for vexxhost ^ | 14:09 |
| lucasagomes | sdague, ok, gimme 1 min I will do it and I ping you back with the results | 14:09 |
| sdague | lucasagomes: cool | 14:09 |
| openstackgerrit | Merged openstack-infra/groups: Add a standalone map page https://review.openstack.org/285943 | 14:09 |
| yolanda | hi mordred, can i get your help on merging some patches that already have a +2? they have been there long | 14:09 |
| yolanda | i can send you the list if you have some minutes | 14:09 |
| *** yaume has quit IRC | 14:12 | |
| *** yaume has joined #openstack-infra | 14:12 | |
| *** doug-fish has joined #openstack-infra | 14:13 | |
| Shrews | mordred: i wonder if there is a cleaner way to handle the bad cloud issue in https://review.openstack.org/285882 | 14:13 |
| Shrews | that solution seems like a fix for only one thing, but maybe we could make it broader across the api somehow | 14:14 |
| Shrews | no broader solution immediately comes to mind though | 14:17 |
| *** exploreshaifali has quit IRC | 14:18 | |
| *** oanson has quit IRC | 14:18 | |
| *** tiswanso has joined #openstack-infra | 14:20 | |
| *** jbernard has left #openstack-infra | 14:20 | |
| *** alivigni has joined #openstack-infra | 14:21 | |
| *** esker has quit IRC | 14:21 | |
| *** mrmartin has quit IRC | 14:21 | |
| mordred | Shrews: yah, I agree on both points | 14:22 |
| *** placko has joined #openstack-infra | 14:22 | |
| *** ildikov has quit IRC | 14:23 | |
| Shrews | mordred: -1'd it so we can take some time discussing/thinking about it | 14:23 |
| *** annegentl_ has quit IRC | 14:23 | |
| *** yamamoto has quit IRC | 14:23 | |
| mordred | kk | 14:23 |
| *** annegentl_ has joined #openstack-infra | 14:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass username/password to SwiftService https://review.openstack.org/287224 | 14:24 |
| mordred | Shrews: there's a follow up to that patch from ianw | 14:24 |
| *** baoli has quit IRC | 14:24 | |
| *** yaume has quit IRC | 14:25 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Raise upper-constraint on python-cinderclient to 1.6.0 https://review.openstack.org/287226 | 14:25 |
| *** yaume has joined #openstack-infra | 14:25 | |
| openstackgerrit | Merged openstack-infra/shade: Invalidate volume cache when waiting for attach https://review.openstack.org/286977 | 14:26 |
| mordred | Shrews: nod on the clarkb patch | 14:27 |
| *** hashar has quit IRC | 14:27 | |
| Shrews | mordred: 'key' is the dict key for password in swift? | 14:29 |
| Shrews | line 742 in your patch | 14:29 |
| *** mriedem has joined #openstack-infra | 14:31 | |
| *** tiswanso has quit IRC | 14:31 | |
| *** baoli has joined #openstack-infra | 14:31 | |
| mordred | Shrews: yah | 14:32 |
| *** mrmartin has joined #openstack-infra | 14:32 | |
| mordred | one sec- lemme doublecheck something | 14:32 |
| *** tiswanso has joined #openstack-infra | 14:33 | |
| mordred | ugh. the url one has a different name | 14:33 |
| mordred | oh! they've got a compat layer. one sec, | 14:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass username/password to SwiftService https://review.openstack.org/287224 | 14:34 |
| mordred | Shrews: there ya go | 14:34 |
| mordred | Shrews: http://git.openstack.org/cgit/openstack/python-swiftclient/tree/swiftclient/service.py#n95 | 14:34 |
| *** baoli has quit IRC | 14:34 | |
| Shrews | ah, that's what i was looking for | 14:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pass username/password to SwiftService https://review.openstack.org/287224 | 14:35 |
| mordred | Shrews: one more update I just saw from reading that | 14:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-mediawiki: Add SmiteSpam extension https://review.openstack.org/287232 | 14:35 |
| mordred | Shrews: it is apparently important to their logic to pass in auth_version | 14:35 |
| *** jsavak has joined #openstack-infra | 14:36 | |
| Shrews | mordred: k. will approve when jenkins says it's ok | 14:37 |
| Shrews | and you stop patching :-P | 14:37 |
| *** baoli has joined #openstack-infra | 14:37 | |
| * Shrews needs moar coffeeeee | 14:37 | |
| *** Swami has joined #openstack-infra | 14:38 | |
| *** ayoung has joined #openstack-infra | 14:39 | |
| mordred | Shrews: I'm done patching :) | 14:40 |
| mordred | Shrews: I think that's all the current outstanding that need landing other than tristan's heat patches | 14:40 |
| mattymo | Can anyone comment on when Summit talks decisions will be finalized? | 14:40 |
| mattymo | in Austin | 14:40 |
| Shrews | mordred: agreed | 14:40 |
| pabelanger | So, wiki.o.o looks good. Day 2, no spamming | 14:40 |
| openstackgerrit | Merged openstack-infra/shade: Also reset swift service object at upload time https://review.openstack.org/286964 | 14:41 |
| pabelanger | now we need to decided how to enable account creation again and remove existing spam | 14:41 |
| pabelanger | 287232 will help with spam | 14:41 |
| mordred | Shrews: and I have a -1/question on the first of those | 14:41 |
| *** hashar has joined #openstack-infra | 14:41 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for volume-backed live migration abort bug 1524898 https://review.openstack.org/286725 | 14:42 |
| openstack | bug 1524898 in OpenStack Compute (nova) "Volume based live migration aborted unexpectedly" [High,Confirmed] https://launchpad.net/bugs/1524898 | 14:42 |
| mordred | pabelanger: awesome job! | 14:42 |
| *** annegentl_ has quit IRC | 14:42 | |
| *** annegentl_ has joined #openstack-infra | 14:43 | |
| *** baoli has quit IRC | 14:44 | |
| *** Liuqing has quit IRC | 14:44 | |
| *** yamamoto has joined #openstack-infra | 14:44 | |
| *** flepied has quit IRC | 14:45 | |
| *** yaume_ has joined #openstack-infra | 14:45 | |
| mriedem | codesearch.o.o isn't available?! | 14:46 |
| *** Swami_ has joined #openstack-infra | 14:46 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Use instance variable in ByteStreamToStreamResult init https://review.openstack.org/286925 | 14:46 |
| mriedem | how will i live? | 14:46 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Remove unused flush from read_subunit() https://review.openstack.org/286928 | 14:47 |
| pabelanger | mriedem: ya, looks down | 14:47 |
| *** gokrokve has joined #openstack-infra | 14:47 | |
| * mordred looking | 14:47 | |
| pabelanger | checking puppetdb to see what is going on | 14:47 |
| *** annegentl_ has quit IRC | 14:47 | |
| pabelanger | puppet seems okay | 14:47 |
| mordred | I have started hound | 14:48 |
| *** baoli has joined #openstack-infra | 14:48 | |
| *** baoli has quit IRC | 14:48 | |
| *** baoli has joined #openstack-infra | 14:48 | |
| mordred | it'll take a few minutes for it to get started up - it does a bunch of git operations | 14:48 |
| *** jsavak has quit IRC | 14:48 | |
| mordred | nm. back up. you should be able to live again mriedem | 14:48 |
| *** rbrndt_ has joined #openstack-infra | 14:49 | |
| mriedem | awesome | 14:49 |
| *** yaume has quit IRC | 14:49 | |
| mriedem | those were dark times | 14:49 |
| mriedem | dark times indeed | 14:49 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add API methods for getting tests by prefix https://review.openstack.org/283334 | 14:49 |
| openstackgerrit | abregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 14:50 |
| *** Swami has quit IRC | 14:50 | |
| *** jsavak has joined #openstack-infra | 14:50 | |
| *** baoli has quit IRC | 14:51 | |
| *** sdake has joined #openstack-infra | 14:51 | |
| *** gokrokve has quit IRC | 14:52 | |
| *** baoli has joined #openstack-infra | 14:52 | |
| *** doug-fish has quit IRC | 14:52 | |
| pabelanger | fungi: if you have time this morning, do you mind checking out 286785 ? It adds experimental bindep jobs for ansible-jobs, which I think is the late group of testing needed for bindep | 14:52 |
| *** doug-fish has joined #openstack-infra | 14:54 | |
| *** baoli has quit IRC | 14:54 | |
| *** pradk has joined #openstack-infra | 14:55 | |
| fungi | pabelanger: why also a -nv suffixed version? | 14:55 |
| *** sshnaidm_ has joined #openstack-infra | 14:56 | |
| jamespage | yolanda, hey - is there any extra setup required todo automatic bug commenting and closure based on gerrit reviews and landing? | 14:56 |
| *** sshnaidm has quit IRC | 14:56 | |
| *** xyang1 has joined #openstack-infra | 14:56 | |
| *** doug-fish has quit IRC | 14:56 | |
| *** sshnaidm_ has quit IRC | 14:56 | |
| *** ildikov has joined #openstack-infra | 14:56 | |
| pabelanger | fungi: mostly forward looking, there are some functional testings that are -nv currently. I figure once this patch is good, I can simply remove the original jobs without having to modify the new bindep ones | 14:56 |
| *** doug-fish has joined #openstack-infra | 14:57 | |
| yolanda | jamespage, you mean, getting -1 from jenkins on tests? | 14:57 |
| fungi | pabelanger: aha, got it. so when this goes live, some projects will be using the -nv version | 14:57 |
| pabelanger | fungi: yup | 14:57 |
| fungi | sensible. thanks! | 14:57 |
| *** zhurong has quit IRC | 14:58 | |
| openstackgerrit | Merged openstack-infra/shade: Use isinstance() for result type checking https://review.openstack.org/286811 | 14:58 |
| *** dmsimard is now known as dmsimard|afk | 14:58 | |
| pabelanger | We'll let it run experimental for a few runs, then setup a time with odyssey4me to replace existing jobs | 14:58 |
| *** gongysh has joined #openstack-infra | 14:58 | |
| fungi | at this point i'm waiting a couple more days for the feature freeze rush to die off before i un-wip the first patch to cut over the python-jobs group to using ubuntu-trusty | 14:58 |
| odyssey4me | fungi ah, the first switchover to ubuntu-trusty... this is going to be fun | 14:59 |
| AJaeger | fungi, for project-config we removed already dvipng and pandoc since there are no users - what do you think of https://review.openstack.org/#/c/286242/ and https://review.openstack.org/#/c/284371/ ? | 14:59 |
| fungi | that'll be the biggest chunk of the transition, and i expect to find most of our corner cases there in lesser-known projects | 14:59 |
| AJaeger | fungi: we could move project-config or manuals to ubuntu-trusty if you want... | 14:59 |
| odyssey4me | we ran it for a while, and lately got a lot more failures than successes - eventually we figured that we'd hold back and stop wasting nodepool resources | 14:59 |
| odyssey4me | pabelanger explain the bindep thing to me? | 15:00 |
| odyssey4me | if you have a moment | 15:00 |
| fungi | pabelanger: yep, getting those out before a transition makes sense, good research | 15:01 |
| fungi | er, AJaeger ^ | 15:01 |
| fungi | not pabelanger | 15:01 |
| pabelanger | odyssey4me: sure, we are moving to -bindep to replace our bare-precise images, which are pinned only to rax. Once on bindep jobs, we'll be using the ubuntu-trusty DIB, across all clouds | 15:02 |
| pabelanger | odyssey4me: TL;DR bare-precise old and busted, ubuntu-trusty new and hotness | 15:02 |
| fungi | odyssey4me: bindep is a tool and mechanism for declaring system-provided dependencies across multiple platforms | 15:02 |
| odyssey4me | ok cool - down with that | 15:02 |
| sgordon | AJaeger, got a second to talk about https://review.openstack.org/#/c/277892/ ? | 15:02 |
| AJaeger | fungi, those were the only obvious extras. I didn't check the rest. | 15:02 |
| AJaeger | sgordon: sure | 15:02 |
| odyssey4me | fungi pabelanger yeah, that's why I was not making the correlation | 15:02 |
| pabelanger | odyssey4me: also allows you to create other-requirements.txt to control which binaries get installed by default | 15:03 |
| jeblair | fungi: it looks like vexxhost is in progress, waiting on image uploads now; is that correct? | 15:03 |
| sgordon | AJaeger, if i understand correctly you are saying some of those skips dont take effect because e.g. - name: gate-functional-dsvm-magnum-api | 15:03 |
| sgordon | branch: ^(?!stable/kilo).*$ | 15:03 |
| sgordon | is later in the file | 15:03 |
| sgordon | ? | 15:03 |
| AJaeger | sgordon: correct | 15:03 |
| sgordon | blergh | 15:03 |
| AJaeger | you might need to reorder these and check the layout output to see whether it does what you want... | 15:03 |
| sgordon | AJaeger, is there anything that checks that isnt the case for others as well? | 15:03 |
| *** sshnaidm has joined #openstack-infra | 15:03 | |
| AJaeger | sgordon: not really ;( This is manually checking | 15:04 |
| sgordon | AJaeger, just because i see after a lot of people glanced over it nobody else noticed this... | 15:04 |
| odyssey4me | pabelanger other-requirements.txt is for the bundep stuff I expect, and we could possible use that when deploying a service from source to reduce our need to carry apt package lists | 15:04 |
| fungi | odyssey4me: the end goal is that we use basically blank slate workers to run all jobs, devstack and tox alike, so this is providing us with a solution to install distro package dependencies for jobs during job runtime, allows individual projects to express their own dependency lists instead of using our fallback default, and allows us to figure out what packages to pre-cache on our workers when we | 15:04 |
| fungi | build images for them | 15:04 |
| sgordon | AJaeger, ok well i will try and fix this one then | 15:04 |
| clarkb | jeblair: basically yes, we ran into some upload fails last night we need to sort out | 15:04 |
| jeblair | clarkb: are we expecting them to fail again? | 15:04 |
| fungi | jeblair: right, what clarkb just said | 15:04 |
| AJaeger | fungi: we could move project-config or manuals to ubuntu-trusty if you want. moving project-config allows us to test it first but if it fails, we really have a problem ;) | 15:04 |
| clarkb | jeblair it seemed consistent last night, was going to ask mnaser about it today | 15:05 |
| mnaser | hi o/ | 15:05 |
| jeblair | clarkb: vexxhost does not use floating ip, correct? | 15:05 |
| clarkb | jeblair: correct | 15:05 |
| fungi | clarkb: your last theory was that they might be limiting concurrent uploads to glance, right? | 15:05 |
| jeblair | pabelanger: dashboard approved! | 15:05 |
| clarkb | fungi: ya because the first upload seemed fine | 15:05 |
| AJaeger | https://review.openstack.org/285722 (project-config) and https://review.openstack.org/283445 (manuals) are the switches to ubuntu-trusty, fungi | 15:06 |
| pabelanger | odyssey4me: right. So one of the things we could do is add a ansible module for bindep. Then leverage other-requirements.txt to handle package dependencies | 15:06 |
| clarkb | and earlier in the day I did manual upload with no problems | 15:06 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs https://review.openstack.org/273424 | 15:06 |
| pabelanger | I plan on creating ansible-role-bindep to play with it | 15:06 |
| mnaser | glance issues at our side? | 15:06 |
| mnaser | (sorry i dont have too much context) | 15:06 |
| clarkb | mnaser: maybe, I can dig out our logs | 15:06 |
| pabelanger | jeblair: \o/ | 15:06 |
| mnaser | small silly question | 15:06 |
| mnaser | how big is the image being uploaded / how long is it taking | 15:06 |
| pabelanger | jeblair: since you are looking at grafana, do you mind looking at 264994 again? Adds some templating support into grafana | 15:07 |
| *** sfinucan has joined #openstack-infra | 15:07 | |
| mnaser | we've seen big images fail to upload because the glance token expires by the time the upload is done so the registry poops out | 15:07 |
| odyssey4me | pabelanger nice idea | 15:07 |
| openstackgerrit | Merged openstack-infra/project-config: Create ansible functional using bindep https://review.openstack.org/286785 | 15:08 |
| *** zz_dimtruck is now known as dimtruck | 15:08 | |
| jeblair | pabelanger: the main thing i'm concerned about is whether it's useful to have the different regions on one graph... tell you what... how about we copy that dashdoard to a new 'experimental' one so we can all look at it and see which we like better? | 15:08 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:08 | |
| pabelanger | jeblair: sure I can do that | 15:08 |
| mnaser | clarkb: i think it might be at our side, im checking | 15:09 |
| *** mtanino has joined #openstack-infra | 15:09 | |
| mnaser | i see a bunch of "Caught error: The client disconnected while sending the POST/PUT body (4286474992 more bytes were expected)" and the request ip is nodepool.openstack.org | 15:09 |
| openstackgerrit | Merged openstack-infra/system-config: Add instructions on how to disable a third party ci account https://review.openstack.org/265359 | 15:09 |
| mnaser | oddly enough they all seem to hover around that amount so im verifying, we have a nginx frontend infront of glance for ssl stuff, checking there | 15:09 |
| *** jsavak has quit IRC | 15:10 | |
| clarkb | mnaser: ok I am just starting my morning so may be slow to respond but can help where I can | 15:11 |
| fungi | it's possible that was our nodepool retrying an upload of the same image, so having it fail at the same bytecount each time is not too strange | 15:11 |
| mnaser | clarkb: np, but i think it might be at your side | 15:11 |
| mnaser | I see it logging the errors as 499 in the nginx logs | 15:11 |
| mnaser | http://stackoverflow.com/a/18410932 | 15:11 |
| jamespage | yolanda, no - so we will reference bugs in the commit entries; nice if we got automatic status setting on the linked bugs... does that make sense? | 15:12 |
| mnaser | and 499 in eror logs implies some timeout at your side possibly | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Create experimental nodepool rackspace dashboard https://review.openstack.org/264994 | 15:12 |
| AJaeger | clarkb, jeblair, fungi: I have a question about one review. training-labs likes to publish tarballs, could you check https://review.openstack.org/#/c/281721/ and tell whether that's the right approach? The change looks fine on technical level but I'd like confirmation on the approach | 15:12 |
| clarkb | mnaser: http://paste.openstack.org/show/488987/ is what shade is logging | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Create experimental nodepool rackspace dashboard https://review.openstack.org/264994 | 15:12 |
| pabelanger | jeblair: experimental ^ | 15:12 |
| AJaeger | jamespage: do you mean http://docs.openstack.org/infra/manual/developers.html#working-on-bugs ? | 15:13 |
| clarkb | mnaser: which is also complaining about connection stuff, so could be | 15:13 |
| * jamespage looks | 15:13 | |
| AJaeger | jamespage: and http://docs.openstack.org/infra/manual/creators.html#create-bug-tracker ? | 15:13 |
| clarkb | mnaser: I think this may be our first glance v2 cloud? so entirely possible this is a bug in our client stuff | 15:14 |
| clarkb | mordred: ^ you may have ideas too | 15:14 |
| jamespage | AJaeger, that's the one - we already have a bug tracker but its not under https://bugs.launchpad.net/<projectname> | 15:14 |
| mnaser | clarkb: v1 is available if you want to fallback to it, but it seems every single upload from nodepool has failed | 15:14 |
| jamespage | its https://bugs.launchpad.net/charms/+source/<projectname [sans charm-]>. | 15:14 |
| mnaser | (from what the logs show) | 15:14 |
| *** _amrith_ is now known as amrith | 15:14 | |
| clarkb | mnaser: ya I don't see any successful ones on my side | 15:14 |
| mordred | clarkb: shouldn't be our first glance v2 cloud | 15:15 |
| jamespage | AJaeger, for example - https://bugs.launchpad.net/charms/+source/nova-compute | 15:15 |
| mordred | clarkb: s/shouldn't/isn't/ | 15:15 |
| mordred | clarkb: but looking | 15:15 |
| fungi | AJaeger: on 281721 i need to check whether $ZUUL_BRANCH is actually set in post. for some reason i thought it wasn't | 15:15 |
| AJaeger | jamespage: no idea whether we support this, hope others can answer... | 15:15 |
| jamespage | ok | 15:15 |
| *** esikache1 has quit IRC | 15:15 | |
| *** austin81 has joined #openstack-infra | 15:16 | |
| mnaser | actually, i see a few images that successfully uploaded | 15:16 |
| *** jaosorior has quit IRC | 15:16 | |
| mnaser | 3x 401s and 4x 204s (which is an a-ok for post-upload) | 15:16 |
| clarkb | mnaser: oh I misread my output | 15:16 |
| clarkb | yup we have 3/4 images "ready" | 15:16 |
| mnaser | also our tokens are set to 6 hours | 15:17 |
| *** yamamoto has quit IRC | 15:17 | |
| *** jaypipes has joined #openstack-infra | 15:17 | |
| AJaeger | fungi: oh, we might need $ZUUL_REFNAME, don't we? | 15:17 |
| *** annegentl_ has joined #openstack-infra | 15:17 | |
| clarkb | http://paste.openstack.org/show/488988/ | 15:17 |
| clarkb | so it does eventually work? | 15:17 |
| *** infra-red has joined #openstack-infra | 15:17 | |
| *** yamamoto has joined #openstack-infra | 15:18 | |
| *** annegentl_ is now known as annegentle | 15:18 | |
| mnaser | hm im not sure if thos eare succesful uploads | 15:18 |
| yolanda | jamespage, sorry but i cannot help with that, other infra-root can help? | 15:18 |
| mnaser | considering that after the 401, the 204 for the same id executed in a few seonds | 15:18 |
| clarkb | mnaser: it isn't attempting to boot them yet so possible that it isn't | 15:18 |
| *** ajmiller has joined #openstack-infra | 15:18 | |
| sdague | clarkb: is there a fix up for openstack health? | 15:18 |
| clarkb | sdague: ? | 15:18 |
| *** yamamoto has quit IRC | 15:18 | |
| sdague | because we've got an 8 hour gate delay, and openstack health hasn't indexed anyting in 2 weeks | 15:19 |
| sdague | so it's hard to understand | 15:19 |
| mnaser | clarkb: logs for file uploads at our side - http://paste.openstack.org/show/488990/ | 15:19 |
| clarkb | sdague: I think mtreinish had subunit parsing bugs that he and pleia2 got sorted out yesterday | 15:19 |
| clarkb | sdague: but I haven't been able to keep up with that | 15:19 |
| sdague | ok | 15:19 |
| sdague | mtreinish is there a thing ready to go? | 15:20 |
| clarkb | mnaser: I am going to manually boot off of those images which should tell us if they worked | 15:20 |
| mnaser | clarkb: okay, and if you can see if a manual glance image-upload works or not, would be helpful in figuring out if there is any network issues | 15:20 |
| mtreinish | sdague: I'm still grasping at straws trying to figure out why the subunit worker isn't working | 15:20 |
| jeblair | jamespage, yolanda, AJaeger: see the 'groups' setting described here: http://docs.openstack.org/infra/manual/creators.html#add-the-project-to-the-master-projects-list | 15:21 |
| *** yamamoto has joined #openstack-infra | 15:21 | |
| jeblair | mtreinish: is there a thing you need me to do? | 15:21 |
| clarkb | mnaser: I do know a manual `openstack image create` worked yesterday bu twith a smaller image | 15:21 |
| mtreinish | jeblair: I'm not sure, I've got a full log from the worker. We're consistently hitting: http://paste.openstack.org/show/488878/ | 15:21 |
| mnaser | yeah maybe it's a size thing or a timeout thing, but removing nodepool from the equation might be helpful to see whats going on | 15:22 |
| mtreinish | but I can't figure out where the file is being closed | 15:22 |
| *** kushal has quit IRC | 15:22 | |
| jamespage | jeblair, looking | 15:22 |
| AJaeger | jeblair: you're right - jamespage you use https://bugs.launchpad.net/charms/, set groups to charm and it should work fine, shouldn't it? | 15:22 |
| clarkb | mnaser: yup | 15:23 |
| lucasagomes | sdague, here it goes https://review.openstack.org/#/c/287216/ | 15:23 |
| openstackgerrit | Rick Salevsky proposed openstack-infra/jenkins-job-builder: Fix disabled condition in case disabled is a string https://review.openstack.org/287257 | 15:23 |
| *** baoli has joined #openstack-infra | 15:23 | |
| lucasagomes | sdague, if you look at the pxe_ipmitool-nv job it's now using ipmitool (w00t) http://logs.openstack.org/16/287216/1/check/gate-tempest-dsvm-ironic-pxe_ipmitool-nv/442affc/logs/screen-ir-cond.txt.gz#_2016-03-02_14_48_47_858 | 15:24 |
| *** jcooley_ has quit IRC | 15:24 | |
| sdague | lucasagomes: great, thanks for that | 15:24 |
| sdague | +2 | 15:24 |
| lucasagomes | sdague, thank YOU! | 15:25 |
| openstackgerrit | Merged openstack-infra/system-config: Add troubleshooting section to github https://review.openstack.org/281726 | 15:25 |
| clarkb | mnaser: 949d5f5e-82e2-4cb0-bad6-91a8883e48a5 doesn't ping or ssh and booted off of one of those images so I think maybe the image isn't good | 15:25 |
| *** ccrouch has joined #openstack-infra | 15:25 | |
| lucasagomes | sorry for bothering, it's because we really want to enable it in Ironic... the ipmitool is our reference driver so being able to test it in gate is a good thing | 15:25 |
| clarkb | (security groups should be open I have been able to ssh and ping other hosts in that project with the same security groups) | 15:25 |
| lucasagomes | "test what you ship, ship what you test" kinda thing :-) | 15:25 |
| *** jsavak has joined #openstack-infra | 15:25 | |
| pabelanger | AJaeger: if you have a moment, 264994 | 15:25 |
| clarkb | mnaser: mordred the error we log on our side is interesting "Error finding address for https://...: Unable to establish connection to https://... | 15:27 |
| jamespage | AJaeger, jeblair: I'll raise a change todo that - might work although the individual project name might still mismatch due to the leading charm- on the project | 15:27 |
| clarkb | mnaser: mordred almost implies name lookup problems, I did confirm that I could establish tcp connections to that host from nodepool.o.o last night fwiw | 15:27 |
| *** woodster_ has joined #openstack-infra | 15:27 | |
| *** jamesmcarthur has joined #openstack-infra | 15:27 | |
| openstackgerrit | Merged openstack-infra/system-config: Add infracloud runbook to regenerate images https://review.openstack.org/283856 | 15:28 |
| openstackgerrit | Merged openstack-infra/system-config: Remove obsolete pandoc install https://review.openstack.org/284371 | 15:28 |
| openstackgerrit | Merged openstack-infra/system-config: Remove obsolete dvipng install https://review.openstack.org/286242 | 15:28 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer https://review.openstack.org/278987 | 15:28 |
| *** baoli has quit IRC | 15:28 | |
| openstackgerrit | Grant Murphy proposed openstack-infra/project-config: Add bandit gate job for Octavia https://review.openstack.org/286912 | 15:29 |
| jeblair | jamespage, AJaeger: i thought in the past we had used it to make a mapping -- it's the most badly named parameter ever, it doesn't actually have anything to do with groups. | 15:29 |
| jeblair | jamespage, AJaeger: it may be worth a look at the jeepyb source to see what it actually does | 15:29 |
| jeblair | jamespage, AJaeger: and other examples in projects.yaml | 15:29 |
| mordred | clarkb: yah - I agree - it seems to be normal requests things rather than anything really related to openstack services | 15:29 |
| *** jsavak has quit IRC | 15:29 | |
| *** jsavak has joined #openstack-infra | 15:30 | |
| AJaeger | jamespage: can't help you with that due to time, do you need more pointers? | 15:30 |
| *** roxanaghe has joined #openstack-infra | 15:30 | |
| *** gordc has joined #openstack-infra | 15:31 | |
| *** baoli has joined #openstack-infra | 15:31 | |
| clarkb | mordred: do you think it is worth setting image_api_version: "1" ? | 15:31 |
| clarkb | to see if there is a difference in behavior between api versions | 15:32 |
| jamespage | AJaeger, I think I have enough to poke around and figure things out | 15:32 |
| mordred | clarkb: I do not | 15:32 |
| mordred | clarkb: I have used shade to upload images to vexxhost using version 2 - also we use version 2 at ovh | 15:33 |
| AJaeger | I have one more question for the experts: https://review.openstack.org/#/c/286956/ enhances the global skip regex and I wonder whether the job isn't just named wrongly. Do we have a naming convention that I'm not aware off? Should it be gate-networking-midonet-v2-dsvm-rally or gate-rally-dsvm-networking-midonet-v2 ? | 15:33 |
| AJaeger | jamespage: Great! | 15:33 |
| mordred | clarkb: which is not helpful in that it does not explain what the heck is going on | 15:34 |
| mordred | clarkb: but it sure does seem like it's getting name resolution errors or something else | 15:34 |
| clarkb | but v2 should work with large images across oceans | 15:34 |
| mordred | yes | 15:34 |
| Zara | email patches are still waiting in review, so I'm here to bug everyone! :D https://review.openstack.org/#/c/270329/ and https://review.openstack.org/#/c/270331/ | 15:34 |
| clarkb | ok, I need to do breakfast and feed kids and stuff will pick this back up after, but please don't stop investigating if you are able :) | 15:34 |
| mordred | if it broke, I would expect it to break with the server dying in the middle of transit | 15:34 |
| *** baoli has quit IRC | 15:35 | |
| jeblair | mtreinish: can you reproduce locally? could there have been a testtools change that alters how it works with stream files? | 15:36 |
| *** dizquierdo has joined #openstack-infra | 15:36 | |
| clarkb | mordred: mnaser also we will want to clean up our image list (there are a bunch that are "queued") | 15:36 |
| clarkb | I will use openstack client to upload after breakfast to at least see if that works with our images | 15:37 |
| mtreinish | jeblair: I asked lifeless about that and he said subunit doesn't close the file | 15:37 |
| mtreinish | jeblair: I'm working on spinning things up locally right now to try and reproduce it | 15:37 |
| *** baoli has joined #openstack-infra | 15:38 | |
| *** piet has quit IRC | 15:39 | |
| *** harlowja_at_home has joined #openstack-infra | 15:40 | |
| mordred | clarkb: well, actually, instead of doing that | 15:40 |
| *** yamahata has joined #openstack-infra | 15:40 | |
| mordred | clarkb: when you get back, can we upload it using shade by hand with debug logging turned on? | 15:40 |
| clarkb | mordred: sure, you can go ahead and start that without me | 15:40 |
| mordred | clarkb: because if that succeeds but nodepool fails, then we know there is somethign extra bonghits goin on, but if it fails, we can get debug logs | 15:40 |
| mordred | clarkb: cool. which image are we uploading? | 15:40 |
| *** korzen has quit IRC | 15:40 | |
| *** abregman has quit IRC | 15:40 | |
| AJaeger | jeblair, fungi: coming back to fungi's comment - after grepping around, I think we need ZUUL_REFNAME instead of ZUUL_BRANCH for the post job of 281721. Do you agree that's better? | 15:40 |
| *** sorantis has quit IRC | 15:41 | |
| fungi | AJaeger: i've just gotten to where i can dig deeper on that one. a moment | 15:41 |
| clarkb | mordred any of fedora-23, devstack-centos7, ubuntu-trusty, or devstack-trusty qcow2 | 15:41 |
| mriedem | AJaeger: it looks like this change only applies to master branches, is that correct? https://review.openstack.org/#/c/286180/1/zuul/layout.yaml | 15:41 |
| mriedem | or by design? | 15:42 |
| *** infra-red has quit IRC | 15:42 | |
| AJaeger | mriedem: it shouldn't | 15:42 |
| clarkb | mordred: the /var/log/nodepool/builder-debug.log should give you extact image filepsths | 15:42 |
| fungi | AJaeger: an example https://jenkins.openstack.org/job/cinder-upstream-translation-update/1588/parameters/ | 15:42 |
| mriedem | AJaeger: oh nvm | 15:42 |
| jeblair | AJaeger: oh yeah probably so | 15:42 |
| mriedem | the change i was watching in liberty wasn't a test change, ignore me | 15:42 |
| *** yamamoto has quit IRC | 15:42 | |
| AJaeger | fungi: so, you're right. thanks. | 15:42 |
| *** gokrokve has joined #openstack-infra | 15:42 | |
| fungi | AJaeger: so, yes, there is no ZUUL_BRANCH provided in post, just ZUUL_REFNAME and ZUUL_REF but i think we've decided previously that the first of those is what we should match on | 15:42 |
| *** yamamoto has joined #openstack-infra | 15:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 15:43 |
| AJaeger | fungi, jeblair: I just pushed using REFNAME ^ | 15:43 |
| *** baoli has quit IRC | 15:43 | |
| *** piet has joined #openstack-infra | 15:43 | |
| *** baoli has joined #openstack-infra | 15:43 | |
| AJaeger | fungi, we currently use both REF and REFNAME - time to cleanup? | 15:43 |
| *** yamamoto has quit IRC | 15:43 | |
| *** dizquierdo has quit IRC | 15:43 | |
| fungi | AJaeger: where is ZUUL_REF being used as a branch name? | 15:44 |
| *** yamamoto has joined #openstack-infra | 15:44 | |
| fungi | AJaeger: there are legitimate reasons to use one or the other, but in post they happen to have the same value | 15:45 |
| AJaeger | jenkins/jobs/requirements.yaml: export ZUUL_REF={branch-override} | 15:45 |
| openstackgerrit | Diana Whitten proposed openstack/requirements: Bump django-compressor to 2.0 https://review.openstack.org/286663 | 15:45 |
| fungi | AJaeger: i think that's because something's using ZUUL_REF so the script is overriding its content, but i'll have to look closer at that job definition | 15:45 |
| AJaeger | jenkins/scripts/propose_update.sh uses also ZUUL_REF | 15:46 |
| fungi | yeah, that seems to be the script that job is calling | 15:46 |
| *** gokrokve has quit IRC | 15:47 | |
| AJaeger | fungi: indeed ;( | 15:47 |
| fungi | AJaeger: i agree, seems to be using it to determine the branch name. we only run that script in post? | 15:47 |
| fungi | i think that probably should switch (gracefully) to ZUUL_REFNAME at some point, though doing so will require a minimum of two patches | 15:48 |
| fungi | temporarily either the job needs to set both or the script needs to use either, then a second patch after image updates can clean out the old variable name | 15:49 |
| *** placko has quit IRC | 15:49 | |
| *** keedya has joined #openstack-infra | 15:49 | |
| clarkb | mordred any luck finding an image file? | 15:49 |
| AJaeger | yes, only in post. I see - will do later... | 15:49 |
| *** fhubik is now known as fhubik_brb | 15:49 | |
| mordred | clarkb: there is a shade upload running in a screen | 15:49 |
| *** esikache1 has joined #openstack-infra | 15:49 | |
| fungi | AJaeger: though also it seems relatively low priority for now since it's working | 15:50 |
| clarkb | mordred: awesome | 15:50 |
| mordred | clarkb: (maybe I should put what I just did in the repl into a little script in case people need to do something similar for debugging) | 15:50 |
| *** Sukhdev has joined #openstack-infra | 15:51 | |
| AJaeger | thanks, fungi. | 15:51 |
| *** jistr has quit IRC | 15:51 | |
| jeblair | mtreinish: if i'm following subunit2sql correctly, it gets jobs as fast as it can, decompresses the subunit file, and puts the entire file contents onto an internal queue. then a second thread pops items off of that queue and processes them, right? | 15:51 |
| *** sdake_ has joined #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack-infra/shade: os_router playbook cleanup https://review.openstack.org/286770 | 15:52 |
| *** sdake has quit IRC | 15:52 | |
| mtreinish | jeblair: right | 15:52 |
| fungi | jeblair: just to confirm, we had previously determined that for jobs in ref-updated pipelines, if we need to identify the name of the branch we should prefer ZUUL_REFNAME instead of ZUUL_REF for that, right? making sure i haven't gotten them backwards | 15:52 |
| *** jpr has joined #openstack-infra | 15:53 | |
| *** gokrokve has joined #openstack-infra | 15:53 | |
| *** gokrokve has quit IRC | 15:53 | |
| *** gokrokve has joined #openstack-infra | 15:53 | |
| jeblair | mtreinish: that will mean scaling this (if it ever becomes necessary) may be difficult because the system may consume jobs faster than it can process them (but those jobs won't be available for other workers)... but perhaps more relevent to the current situation.... | 15:53 |
| jeblair | mtreinish: i don't think we can tell which subunit stream it bombed on | 15:54 |
| jeblair | mtreinish: (i'm also slightly worried about the memory consumption of a whole lot of cstringio objects waiting in memory to be processed) | 15:54 |
| mtreinish | jeblair: well, I stole the approach from the logstash processor :) | 15:54 |
| mtreinish | jeblair: and speaking of memory consumption it crashed because of oomkiller 2 weeks ago | 15:55 |
| *** sdake has joined #openstack-infra | 15:55 | |
| jeblair | mtreinish: i think the logstash processor may wait until its internal queue item is consumed before reporting the job as complete... let me check that | 15:55 |
| *** gongysh has quit IRC | 15:56 | |
| mtreinish | jeblair: I added the str(stringio) (which has the memory addr for the object) to the log so we can see when something gets pushed on the queue and when the processor pulls it off | 15:56 |
| *** fhubik_brb is now known as fhubik | 15:57 | |
| *** eric_lopez has quit IRC | 15:57 | |
| *** piet has quit IRC | 15:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add debug message about file hash calculation https://review.openstack.org/287281 | 15:57 |
| mordred | clarkb: ^^ just added that so that it's clear when shade is spinning on hash calculation | 15:58 |
| *** sdake_ has quit IRC | 15:58 | |
| AJaeger | I have one more question for the experts: https://review.openstack.org/#/c/286956/ enhances the large skip-if regex and I wonder whether the job isn't just named wrongly. Do we have a naming convention that I'm not aware off? Should it be gate-networking-midonet-v2-dsvm-rally or gate-rally-dsvm-networking-midonet-v2 ? fungi, jeblair, clarkb, can either of you answer on the naming here, please? | 15:58 |
| jeblair | mtreinish: you are right about the log processor, it also has two disconnected processing threads, though it seems that most of the processing is in the gearman-synced thread, so the other shouldn't get too far behind. nonetheless, i bet we can make subunit simpler... | 15:58 |
| *** flepied has joined #openstack-infra | 15:59 | |
| *** baoli has quit IRC | 15:59 | |
| fungi | AJaeger: yeah, saw your question earlier--was going to ponder that one next | 15:59 |
| *** esikache1 has quit IRC | 15:59 | |
| jeblair | mtreinish: ah, so we should be able to map an error to a specific subunit file? | 15:59 |
| AJaeger | fungi, sorry for my impatience ;( | 15:59 |
| fungi | AJaeger: no, it's fine, thanks for the reminder | 15:59 |
| *** baoli has joined #openstack-infra | 16:00 | |
| *** maishsk has quit IRC | 16:00 | |
| mtreinish | jeblair: yeah, that was my intent, I put the full log here: http://blog.kortar.org/wp-content/uploads/2016/02/subunit-worker-A-debug.log2 | 16:01 |
| fungi | AJaeger: so basically if "rally" appears SOMEWHERE before "-dsvm" in the job name, then they don't need to adjust the regex | 16:01 |
| jeblair | mtreinish: so if i'm following you and the code correctly, the example in http://paste.openstack.org/show/488878/ really did bomb on the file listed at the top because the mem addresses of the 2 stringio log lines match? | 16:01 |
| *** rhallisey has quit IRC | 16:01 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Speed up tox test count https://review.openstack.org/287285 | 16:01 |
| mnaser | clarkb: i'm pretty confused about that error to be honest | 16:02 |
| *** piet has joined #openstack-infra | 16:02 | |
| mtreinish | jeblair: yep, that's what it looks like | 16:02 |
| mtreinish | and that's where I get very confused :) | 16:02 |
| mnaser | it seems to imply that the connection is being dropped or it's unable to communicate | 16:02 |
| *** rhallisey has joined #openstack-infra | 16:02 | |
| AJaeger | fungi, it needs to be rally-dsvm | 16:02 |
| jeblair | mtreinish: then hrm! i can't reproduce locally, even after a local revert of your removal of the flush call. | 16:02 |
| pabelanger | AJaeger: looks like gate failure on 287211 | 16:02 |
| jeblair | mtreinish: this is weird | 16:02 |
| pabelanger | AJaeger: git clone issue | 16:02 |
| jeblair | mtreinish: i'll see if breakfast helps. | 16:03 |
| *** matrohon has quit IRC | 16:03 | |
| fungi | AJaeger: oh, yep you're correct. there's no wildcarding between those | 16:03 |
| AJaeger | pabelanger: please recheck it | 16:03 |
| mtreinish | jeblair: heh, I slept on it and I'm still not any closer :) | 16:03 |
| pabelanger | AJaeger: ah, I wasn't sure that would help. Doing so now | 16:04 |
| AJaeger | fungi: So, my question is: Is there any kind of naming that we should try to suggest/enforce - or is it frree for all | 16:04 |
| fungi | AJaeger: so it could be gate-networking-midonet-v2-rally-dsvm-nv if they wanted | 16:04 |
| mnaser | clarkb: if you can manually upload a nodepool image, that'd help in narrowing it down (unless you're still getting that error when image-create too) | 16:04 |
| *** rhallisey has quit IRC | 16:04 | |
| AJaeger | fungi: yes | 16:04 |
| mtreinish | jeblair: it really looks like it's reusing that first subunit stringio object. Looking at the full log the first 1 works and then all the others after fail | 16:04 |
| clarkb | mnaser: mordred has manual uplaod with all the debug logging running now | 16:04 |
| *** tiswanso has quit IRC | 16:04 | |
| *** pcaruana has quit IRC | 16:05 | |
| mnaser | cool | 16:05 |
| clarkb | mnaser hopefully that clears it up | 16:05 |
| mtreinish | jeblair: but, I just don't know how that could be happening | 16:05 |
| *** rhallisey has joined #openstack-infra | 16:05 | |
| fungi | AJaeger: i would like to see us shrink that regex if possible, but also the proliferation of skip-if rules for each individual project is getting insane | 16:05 |
| mnaser | clarkb: i secretly hope it doesnt work, because that'll be easier to troubleshoot than something involving nodepool :-p | 16:05 |
| clarkb | ya | 16:05 |
| fungi | AJaeger: so pushing back on additions like this may just be wasted effort | 16:05 |
| *** whoops has quit IRC | 16:06 | |
| AJaeger | fungi, yeah - seems each project is adding one now | 16:06 |
| *** fawadkhaliq has joined #openstack-infra | 16:06 | |
| eharney | could someone help me figure out why https://review.openstack.org/#/c/264982/ doesn't seem to be getting into the gate queue after check passes? | 16:06 |
| fungi | AJaeger: it may be easier to not object to additions which are technically correct, and look forward to this all ending up in the individual projects' repos come zuul v3 | 16:06 |
| *** fhubik is now known as fhubik_brb | 16:07 | |
| fungi | AJaeger: so providing them with more attractive solutions is fine, but we've got other battles where our time would be better spent | 16:07 |
| mordred | mnaser, clarkb: manual upload using shade worked fine | 16:07 |
| * mnaser wonders if concurrency is an issue | 16:07 | |
| AJaeger | fungi: yes, that outlook sounds fine. Ok, will review the change later and +2 (didn't object to it so far, just commented) | 16:08 |
| mnaser | i can't imagine it being one though | 16:08 |
| AJaeger | fungi: Thanks! | 16:08 |
| *** smarcet has joined #openstack-infra | 16:08 | |
| *** esker has joined #openstack-infra | 16:08 | |
| *** piet has quit IRC | 16:08 | |
| *** jaypipes has quit IRC | 16:08 | |
| *** pots79 has joined #openstack-infra | 16:08 | |
| fungi | eharney: why would that go back into the gate again? | 16:09 |
| *** tiswanso has joined #openstack-infra | 16:09 | |
| *** esker has quit IRC | 16:09 | |
| nibalizer | good morning | 16:09 |
| *** whoops has joined #openstack-infra | 16:09 | |
| mordred | clarkb: maybe we should figure out a way to turn on the http_debug logging for just vexxhost while running in nodepool and log just that to a diffrent file | 16:09 |
| fungi | eharney: i'm confused by what you expect from retesting changes that have been merged for days | 16:09 |
| mnaser | mordred, clarkb: could you kick off a reupload of images via nodepool to see if the issue repeats itself? maybe there was some network issues this morning (as far as i know, there shouldn't be :X) | 16:09 |
| eharney | fungi: whoops, i pasted the wrong link, i meant https://review.openstack.org/#/c/264983/ | 16:09 |
| *** esker has joined #openstack-infra | 16:09 | |
| *** _nadya_ has quit IRC | 16:09 | |
| *** BigWillie has joined #openstack-infra | 16:10 | |
| mordred | (I'm going to delete the image I just uploaded) | 16:10 |
| *** armax has joined #openstack-infra | 16:11 | |
| *** sdake_ has joined #openstack-infra | 16:12 | |
| *** daemontool_ has joined #openstack-infra | 16:12 | |
| *** daemontool has quit IRC | 16:13 | |
| *** exploreshaifali has joined #openstack-infra | 16:13 | |
| *** rockyg has joined #openstack-infra | 16:13 | |
| *** ikalnitsky_ has joined #openstack-infra | 16:13 | |
| *** fawadkhaliq has quit IRC | 16:14 | |
| *** mwhahaha_ has joined #openstack-infra | 16:14 | |
| *** bauzas_ has joined #openstack-infra | 16:14 | |
| fungi | eharney: it depends-on a cinder change which didn't merge until after this one was approved, and apparently cinder and cinderclient no longer share any jobs so are not gated in a shared change queue | 16:14 |
| fungi | eharney: you'll need to add or replace the workflow +1 on 264983 to get zuul to notice it's ready for gating now | 16:15 |
| *** odyssey4me_ has joined #openstack-infra | 16:15 | |
| *** bryan_att has quit IRC | 16:15 | |
| *** mwhahaha has quit IRC | 16:15 | |
| *** larrie_kong has quit IRC | 16:15 | |
| *** niska has quit IRC | 16:15 | |
| *** mgkwill has quit IRC | 16:15 | |
| *** ikalnitsky has quit IRC | 16:15 | |
| *** bauzas has quit IRC | 16:15 | |
| *** larrie_kong has joined #openstack-infra | 16:15 | |
| *** thorst_ has quit IRC | 16:15 | |
| *** maximov_ has quit IRC | 16:15 | |
| *** gnuoy has quit IRC | 16:15 | |
| *** odyssey4me has quit IRC | 16:15 | |
| *** briancurtin has quit IRC | 16:15 | |
| *** gnuoy has joined #openstack-infra | 16:15 | |
| *** johnthetubaguy has quit IRC | 16:15 | |
| *** smarcet has quit IRC | 16:15 | |
| *** sdake has quit IRC | 16:15 | |
| *** markmcclain has quit IRC | 16:15 | |
| *** msuriar has quit IRC | 16:15 | |
| *** sulo has quit IRC | 16:15 | |
| *** alaski has quit IRC | 16:15 | |
| *** bauzas_ is now known as bauzas | 16:15 | |
| *** alaski_ has joined #openstack-infra | 16:15 | |
| *** kencjohnston has joined #openstack-infra | 16:15 | |
| *** johnthetubaguy has joined #openstack-infra | 16:15 | |
| *** larrie_kong has quit IRC | 16:15 | |
| *** larrie_kong has joined #openstack-infra | 16:15 | |
| *** markmcclain has joined #openstack-infra | 16:15 | |
| *** mwhahaha_ is now known as mwhahaha | 16:15 | |
| *** msuriar has joined #openstack-infra | 16:15 | |
| *** ikalnitsky_ is now known as ikalnitsky | 16:16 | |
| eharney | fungi: ah ok, thanks | 16:16 |
| *** thorst_ has joined #openstack-infra | 16:16 | |
| *** smarcet has joined #openstack-infra | 16:16 | |
| *** briancurtin has joined #openstack-infra | 16:16 | |
| *** niska has joined #openstack-infra | 16:17 | |
| *** sulo has joined #openstack-infra | 16:17 | |
| *** alaski_ is now known as alaski | 16:17 | |
| fungi | eharney: basically that's a failsafe, since the lack of any shared integration testing between cinder and cinderclient means that if the cinder change broke this cinderclient change the only way you'll have to know is after the cinder change has merged | 16:17 |
| *** maximov_ has joined #openstack-infra | 16:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add simple script to help debug shade image uploads https://review.openstack.org/287296 | 16:17 |
| mordred | clarkb: ^^ there is a script version of what I did in the repl earlier | 16:17 |
| *** mattt has left #openstack-infra | 16:17 | |
| eharney | fungi: is this a new failsafe, or it just happened due to reconfiguration of some jobs? (just curious) | 16:18 |
| AJaeger | eharney: http://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats | 16:18 |
| fungi | eharney: it's how the feature has always worked. basically depends-on has two different uses: for projects sharing a change queue it means test the change with the dependency applied. for projects without a shared change queue it means don't land this change until after the declared dependency merges | 16:19 |
| * AJaeger will be back later... | 16:19 | |
| *** bryan_att has joined #openstack-infra | 16:19 | |
| eharney | fungi: i guess i always was thinking of the latter case | 16:20 |
| fungi | eharney: if cinder and/or cinderclient recently removed some jobs such that they ceased having any in common, then depends-on behavior will also have changed between them accordingly | 16:20 |
| eharney | right | 16:20 |
| *** infra-red has joined #openstack-infra | 16:20 | |
| mtreinish | jeblair: just for a sanity check is there anything alarming in dmesg on subunit-worker01.o.o? | 16:20 |
| jeblair | mtreinish: i think i see the problem; patch in progress | 16:21 |
| *** sdake_ is now known as sdake | 16:21 | |
| *** mgkwill has joined #openstack-infra | 16:21 | |
| fungi | mtreinish: oom killer | 16:22 |
| fungi | [Tue Mar 1 01:00:38 2016] Killed process 16382 (subunit-gearman) total-vm:17108324kB, anon-rss:7615884kB, file-rss:0kB | 16:22 |
| *** jaypipes has joined #openstack-infra | 16:22 | |
| *** Swami__ has joined #openstack-infra | 16:22 | |
| mtreinish | fungi: that's surprising, because it shouldn't have been running | 16:23 |
| mrmartin | fungi, pabelanger: I sucessfully deployed a wiki.o.o clone into a dev machine, but it was not trivial, lot's of things are missing from the puppet and .gitmodules are not up-to date. | 16:24 |
| mtreinish | jeblair: cool | 16:24 |
| fungi | mrmartin: yes, i think the volunteer who originally set up that server did a lot of it manually with the intent of getting more of it into puppet, but never found the time. it's not a great situation | 16:25 |
| pabelanger | mrmartin: Ya, I expected that. I haven't gotten around to testing wiki-dev.o.o on trusty yet | 16:25 |
| fungi | we've gotten a lot stricter in years since about making sure deployment starts in puppet rather than being automated after teh fact | 16:25 |
| mrmartin | it is on precise actually, I'll share the deployment memo somewhere | 16:26 |
| openstackgerrit | Merged openstack-infra/shade: Pass username/password to SwiftService https://review.openstack.org/287224 | 16:26 |
| mrmartin | i made a round with master but it have a different deployment model, and uses composer for modules | 16:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/subunit2sql: Don't extend the passed in targets https://review.openstack.org/287299 | 16:26 |
| jeblair | mtreinish: ^ | 16:26 |
| *** Swami_ has quit IRC | 16:26 | |
| jeblair | mtreinish: your observation about the first one succeeding but subsequent ones failing, and also eating breakfast, led me to that ^ | 16:27 |
| jeblair | mtreinish: i was able to reproduce it locally and verify the fix | 16:27 |
| mrmartin | the 1.25 version of wiki.o.o was released in 2014, so the core and extensions were not updated since then. | 16:27 |
| mtreinish | jeblair: oh, that's a good catch | 16:27 |
| pabelanger | mrmartin: 285405 was my entrypoint, but like I said, haven't tested it locally. | 16:27 |
| jeblair | mtreinish: here's my test program: http://paste.openstack.org/show/489002/ | 16:27 |
| * mtreinish curses SpamapS and his targets api :) | 16:28 | |
| jeblair | mtreinish: and the commented out line in the middle toggles fixed/broken behavior | 16:28 |
| mrmartin | ok, as a next step I want to check the new user registration / captcha story, if we can solve that, I guess we can turn back the user registration. | 16:28 |
| mrmartin | the openid one | 16:28 |
| SpamapS | I have a targets API? | 16:28 |
| *** gongysh has joined #openstack-infra | 16:28 | |
| mtreinish | SpamapS: https://review.openstack.org/#/c/287299/1 | 16:28 |
| mtreinish | jeblair: I really want to fast track that fix, but do you think we could add a unit test for this case | 16:29 |
| jeblair | SpamapS: congratulations! it's a targets API! | 16:29 |
| fungi | mrmartin: yeah, i've occasionally popped in and updated mediawiki on that server for patches to critical vulnerabilities when they're in features we're exposing, but it's not been frequent and i've no doubt i've probably missed some less obvious issues | 16:30 |
| mtreinish | SpamapS: heh, I forgot what you called it. But it's the thing you added to add the hooks for appending the counters data to the end of the subunit | 16:30 |
| clarkb | mnaser: ya I can trigger some additional uploads though at this time of the day we will be competing with all of the other uploads so may not be quick | 16:31 |
| jeblair | mtreinish: let me see what i can do about a test | 16:31 |
| mtreinish | jeblair: cool, thanks | 16:32 |
| mtreinish | I just want to make sure we don't regress here again | 16:32 |
| *** esikache1 has joined #openstack-infra | 16:32 | |
| clarkb | mnaser: actually it is still trying to upload that one image over and over and still hitting this so I don't think it is transient network issues | 16:33 |
| *** kushal has joined #openstack-infra | 16:33 | |
| *** odyssey4me_ is now known as odyssey4me | 16:33 | |
| *** jsavak has quit IRC | 16:33 | |
| *** electrofelix has quit IRC | 16:34 | |
| *** tiswanso has quit IRC | 16:34 | |
| *** keedya has quit IRC | 16:34 | |
| *** tiswanso has joined #openstack-infra | 16:34 | |
| *** sdake has quit IRC | 16:35 | |
| SpamapS | mtreinish: yeah. I hope to get back to that soon.... We need it | 16:35 |
| *** david-lyle has quit IRC | 16:36 | |
| *** infra-red has quit IRC | 16:38 | |
| *** david-lyle has joined #openstack-infra | 16:38 | |
| *** eric_lopez has joined #openstack-infra | 16:38 | |
| *** rhedlind has left #openstack-infra | 16:39 | |
| *** yamahata has quit IRC | 16:39 | |
| *** eric_lopez has quit IRC | 16:40 | |
| *** rcarrillocruz has left #openstack-infra | 16:40 | |
| *** apoorvad has joined #openstack-infra | 16:40 | |
| *** sdake has joined #openstack-infra | 16:40 | |
| *** eric_lopez has joined #openstack-infra | 16:40 | |
| *** regXboi has joined #openstack-infra | 16:41 | |
| *** krtaylor has quit IRC | 16:41 | |
| clarkb | mnaser: mordred reading up on the glance v2 api the first step is to create the image which returns the file path, then you PUT to the file path | 16:41 |
| clarkb | mnaser: mordred the error message that is being logged implies the first POST to create the image is succeeding and returning a file path | 16:42 |
| clarkb | mnaser: mordred then for some reason we fail to connect during the PUT | 16:42 |
| *** yamamoto has quit IRC | 16:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add vexxhost mirror to documentation and cacti https://review.openstack.org/287310 | 16:43 |
| *** infra-red has joined #openstack-infra | 16:43 | |
| fungi | clarkb: ^ i assume you didn't have that proposed yet | 16:43 |
| clarkb | (also the images the "succeeded" are not working in a way that I can debug so I don't think we actually succeeded there) | 16:43 |
| clarkb | fungi: I hadn't | 16:43 |
| mordred | clarkb: I agree with that reading | 16:44 |
| *** BigWillie has quit IRC | 16:45 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 16:45 |
| *** jsavak has joined #openstack-infra | 16:45 | |
| openstackgerrit | devdatta-kulkarni proposed openstack/requirements: Adding python-solumclient https://review.openstack.org/287313 | 16:45 |
| *** ildikov has quit IRC | 16:46 | |
| *** sc68cal has joined #openstack-infra | 16:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/subunit2sql: Don't extend the passed in targets https://review.openstack.org/287299 | 16:47 |
| *** esker has quit IRC | 16:47 | |
| jeblair | mtreinish, SpamapS: ^ | 16:47 |
| *** resker has joined #openstack-infra | 16:47 | |
| jeblair | mordred: ^ | 16:47 |
| *** ashleighfarnham has joined #openstack-infra | 16:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/subunit2sql: Don't extend the passed in targets https://review.openstack.org/287299 | 16:47 |
| jeblair | drat; typo in comment^ | 16:48 |
| lucasagomes | clarkb, jeblair if you guys have a second, mind taking a look at https://review.openstack.org/#/c/284036/ ? Thanks a lot | 16:48 |
| clarkb | mordred: mnaser a tcpdump against that dest host shows lots of bits going by which implies a portion of the upload is working | 16:49 |
| *** infra-red has quit IRC | 16:49 | |
| jeblair | clarkb: osic sitrep? | 16:51 |
| *** rlandy has quit IRC | 16:51 | |
| clarkb | jeblair: everything ready to go (now that i have working volume attach launch node stuff) except that their ssl situation is still :/ | 16:51 |
| jeblair | clarkb: that's waiting on them; or should we look into a workaround? | 16:51 |
| clarkb | jeblair: we have to go fully unverified ssl to use the cloud in its current situation due to them not having working dns that maps to cert CNs | 16:51 |
| clarkb | jeblair: the only viable workaround for us is to not verify ssl at all to them | 16:52 |
| fungi | yeah, cert cn there is for a hostname which is not in dns and which they're not going to put in dns. instead they're waiting for intel to add a dns record and they'll replace the cert with one for that new cn | 16:52 |
| jeblair | do we have an idea if that's a days/weeks sort of thing? | 16:53 |
| clarkb | fungi: the dns record is actually there fwiw, they could use a self signed cert for that name in the interim but aren't for some reason | 16:53 |
| mtreinish | jeblair: cool, thanks. Once that lands I'll push a release and we can restart the worker after it gets pulled in by puppet | 16:53 |
| clarkb | jeblair: I asked and haven't heard back | 16:53 |
| jeblair | mtreinish: ++ | 16:53 |
| fungi | whoever thought it was a good idea to have one organization stand up these environments yet have to go through a different organization for the cert and dns record of the api endpoint was not thinking clearly | 16:53 |
| *** rossella_s has quit IRC | 16:53 | |
| fungi | clarkb: oh, the new dns name exists and points to that ip address now? | 16:54 |
| clarkb | fungi: yes cloud1.osic.org has an A record | 16:54 |
| *** rossella_s has joined #openstack-infra | 16:54 | |
| clarkb | aiui they are waiting for the soa to ack the cert request | 16:54 |
| fungi | still, yeah the certificate mismatch is a no-go. i don't care if it's self-signed or not, it's unusable if it doesn't map to the hostname we're connecting to | 16:54 |
| *** esikache1 has quit IRC | 16:54 | |
| *** ttx has quit IRC | 16:54 | |
| *** vgridnev has quit IRC | 16:55 | |
| nibalizer | well yay for getting dns | 16:55 |
| nibalizer | I arm running do-release-upgrade today so this might be the last time you hear from me for a while :P | 16:55 |
| jeblair | cool, so maybe just a matter of a little more waiting? | 16:55 |
| clarkb | jeblair: ya I think its mostly up to them getting the soa to ack the cert, putting cert in place then updating keystone catalog | 16:56 |
| *** ttx has joined #openstack-infra | 16:56 | |
| yolanda | nibalizer, care reviewing some stuff before going into the dark? | 16:56 |
| clarkb | we can use that cloud now if we want to not verify ssl but fungi seemed hesitant to do that | 16:56 |
| jeblair | nibalizer: i did that recently on my laptop -- suddenly in wily the full disk encryption boot prompt *works*! that's so much better than on vivid where it did not! | 16:56 |
| clarkb | which is understandable given there are reasonable workarounds if dns is working | 16:56 |
| fungi | clarkb: i thought you said latest neutronclient didn't support that? | 16:56 |
| fungi | --insecure became a no-op in a recent release | 16:57 |
| clarkb | fungi: only via cli, openstackclient and shade seem fine | 16:57 |
| fungi | i'm mainly leery of saying we're in production on a cloud where we have to downgrade to older client software to interface with it | 16:57 |
| jeblair | fungi, clarkb: i guess we explored the idea of putting something in /etc/hosts but decided not to because that's just bonkers? | 16:57 |
| clarkb | jeblair: I actually did explore that :) it doesn't work because the catalog doesn't use the host name it uses the ip addr | 16:58 |
| clarkb | jeblair: so that works for your first request to get a token and find the next service but fails when you start talking to not keystone | 16:58 |
| fungi | righht, they need to also update the keystone catalog, i forgot about that bit | 16:58 |
| jeblair | clarkb: i'm kind of glad that doesn't work. i'm not entirely sure i'm glad for the reason. :) | 16:58 |
| jeblair | also, i'd really like to get 8 more clouds from them. do you think that's possible? | 16:59 |
| jeblair | (cloud9.osic.org? !) | 17:00 |
| *** _nadya_ has joined #openstack-infra | 17:00 | |
| fungi | hah | 17:00 |
| fungi | you can ask to go back into the waiting list for slot #9 i guess | 17:00 |
| *** vincentll has quit IRC | 17:01 | |
| clarkb | I think in an ideal world they would've use a self signed cert against a name in dns that they control to start | 17:02 |
| clarkb | then we could do like we did with infra cloud | 17:02 |
| jeblair | i'm going to restart statsd so we lose the infra-cloud gauges | 17:02 |
| clarkb | and hwen they get dns sorted for long term switch over | 17:02 |
| *** rcarrillocruz has joined #openstack-infra | 17:02 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Change kolla gate timeout to 90 minutes https://review.openstack.org/287321 | 17:03 |
| *** gyee has joined #openstack-infra | 17:03 | |
| clarkb | mnaser: mordred also I confirmed that vexxhost appears to use dhcp for eth0 configuration which means my servers booted off the image uploads that "succeeded" probably aren't failing due to network setup problems | 17:04 |
| *** rlandy has joined #openstack-infra | 17:04 | |
| *** rhallisey has quit IRC | 17:05 | |
| clarkb | lack of console log seems to confirm | 17:05 |
| *** rhallisey has joined #openstack-infra | 17:06 | |
| mnaser | in a conference call i'll jump in soon :-p | 17:06 |
| *** scheuran has quit IRC | 17:07 | |
| zaro | morning | 17:07 |
| *** abregman has joined #openstack-infra | 17:09 | |
| *** fhubik_brb is now known as fhubik | 17:09 | |
| *** baoli has quit IRC | 17:09 | |
| *** yamahata has joined #openstack-infra | 17:09 | |
| *** kdas_ has joined #openstack-infra | 17:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Create experimental nodepool rackspace dashboard https://review.openstack.org/264994 | 17:10 |
| *** electrofelix has joined #openstack-infra | 17:10 | |
| clarkb | mordred: when you did your upload with shade you did it on nodepool.o.o using system libs? | 17:11 |
| fungi | jenkins01 and 07 are looking potentially unhappy. i'll see what i can do to cheer them up | 17:11 |
| *** baoli has joined #openstack-infra | 17:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Add irc channel for the i18n french team https://review.openstack.org/287048 | 17:11 |
| *** e0ne has quit IRC | 17:11 | |
| anteaya | fungi: thank you | 17:11 |
| anteaya | morning zaro | 17:11 |
| *** jpr has quit IRC | 17:12 | |
| *** pleia2 has joined #openstack-infra | 17:12 | |
| *** Swami__ has quit IRC | 17:12 | |
| *** ildikov has joined #openstack-infra | 17:12 | |
| zaro | jeblair: did you get my msg about trying to fix the gerrit issue where it some changes are not searchable? | 17:12 |
| *** achanda has joined #openstack-infra | 17:12 | |
| *** kushal has quit IRC | 17:12 | |
| *** kdas_ is now known as kushal | 17:13 | |
| *** degorenko is now known as _degorenko|afk | 17:13 | |
| *** kushal has quit IRC | 17:13 | |
| *** kushal has joined #openstack-infra | 17:13 | |
| *** sdake has quit IRC | 17:13 | |
| jeblair | zaro: sorry i missed that | 17:13 |
| clarkb | ok reading the glanceclient code the "Error finding address" is misleading and appears to be a copy pasta thing | 17:14 |
| clarkb | it is right next to where name resolution errors are actually handled | 17:14 |
| *** korzen has joined #openstack-infra | 17:14 | |
| mordred | clarkb: yes | 17:15 |
| korzen | Hi, is there anyone able to get my patch +2? https://review.openstack.org/#/c/250215 I've got +1 from armax, ihrachys and sc68cal, it is the DVR multinode grenade job for Neutron upgrade tests - sdague anteaya | 17:15 |
| mordred | clarkb: used system libs | 17:15 |
| zaro | jeblair: maybe give fix-change a try: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-18.log.html#t2016-02-18T00:23:11 | 17:16 |
| * armax looks | 17:16 | |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| *** rfolco has quit IRC | 17:17 | |
| *** sdake has joined #openstack-infra | 17:17 | |
| sdague | korzen: +2 | 17:17 |
| openstackgerrit | Merged openstack-infra/shade: Fix create_server() with a named network https://review.openstack.org/286787 | 17:17 |
| *** xarses has joined #openstack-infra | 17:17 | |
| korzen | thanks sdague :) | 17:17 |
| clarkb | mordred: your idea for running the builder with verbose shade logging makes sense to me | 17:18 |
| *** jlanoux has quit IRC | 17:18 | |
| *** jpr has joined #openstack-infra | 17:19 | |
| clarkb | mordred: as a next step I think it will provide the most info, not sure how to limit that to a speicfic provider though | 17:19 |
| *** salv-orl_ has joined #openstack-infra | 17:19 | |
| clarkb | maybe we wait for the current builds to other cloud sto finish then restart with that in place | 17:19 |
| mordred | clarkb: ++ | 17:19 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Add experimental Manila LVM job with minimal services https://review.openstack.org/285828 | 17:20 |
| clarkb | mordred: can that be toggled in the log config file easily? | 17:20 |
| *** resker has quit IRC | 17:20 | |
| mordred | clarkb: looking now | 17:21 |
| *** fawadkhaliq has joined #openstack-infra | 17:21 | |
| zaro | clarkb: so we are still having the same jvm gc issue with gerrit? | 17:21 |
| *** salv-orlando has quit IRC | 17:22 | |
| *** marg7175_ has joined #openstack-infra | 17:22 | |
| clarkb | zaro: yes Ithink so. We had to restart last week to handle it | 17:22 |
| *** dizquierdo has joined #openstack-infra | 17:22 | |
| *** jaypipes has quit IRC | 17:22 | |
| mordred | clarkb: tl;dr - we want to enable debug logging for keystoneauth | 17:22 |
| *** nmagnezi has quit IRC | 17:22 | |
| zaro | clarkb: did you confirm it's the same issue? | 17:22 |
| korzen | any second +2 for my patch? | 17:23 |
| mordred | clarkb: so I think that is "yes" | 17:23 |
| clarkb | zaro: we confirmed that GC was high and we were using lots of memory I did not dig in logs to see if it was the same tracebacks | 17:23 |
| clarkb | mordred: just keystoneauth? I think we are failing when glanceclient talks to the glance server | 17:23 |
| zaro | clarkb: were people report 500 errors that prompted to restart? | 17:23 |
| clarkb | zaro: yes and server slowness | 17:23 |
| *** mriedem has left #openstack-infra | 17:24 | |
| *** jcooley_ has joined #openstack-infra | 17:24 | |
| *** mriedem has joined #openstack-infra | 17:24 | |
| *** arxcruz has quit IRC | 17:24 | |
| zaro | clarkb: so last we discussed the plan was to up the memory, we still gonna do that? | 17:24 |
| *** krtaylor has joined #openstack-infra | 17:24 | |
| *** electrofelix has quit IRC | 17:25 | |
| *** xiangxinyong has quit IRC | 17:25 | |
| clarkb | zaro: someone needs to make a plan to do it | 17:25 |
| clarkb | but I think we should | 17:25 |
| clarkb | the biggest problem there is our IP address will change so we need to let people know what the new address is ahead of time | 17:25 |
| clarkb | and we need tofigure out how much bigger we should grow | 17:25 |
| mnaser | clarkb: yup, we have dhcp | 17:26 |
| mordred | clarkb: all http requests use keystoneauth | 17:26 |
| mnaser | so if you don't have a console log, the image is probably dun goof'd | 17:26 |
| mordred | clarkb: I'm doing a patch right now | 17:26 |
| clarkb | mordred: ok | 17:26 |
| zaro | clarkb: does something like this need a spec or can we just discuss/plan in a meeting? | 17:26 |
| clarkb | mnaser: yup | 17:26 |
| mnaser | i'd check the size of the image | 17:26 |
| *** x00350071 has joined #openstack-infra | 17:26 | |
| clarkb | zaro: I don't know that it needs a formal spec, but someone needs to do napkin math to figure out how big we need to go, get an infra root to boot the new server so that IP is assigned, then email that out wtih a cutover date | 17:27 |
| mnaser | clarkb: if the server goes to ACTIVE state as well, retrieving vnc session might be useful to see what/where it's stuck on | 17:27 |
| *** achanda has quit IRC | 17:27 | |
| zaro | clarkb: ok. gerrit-tuning is on the meeting agenda. we can probably discuss with crew and see where it goes. | 17:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add log file for http requests for nodepool https://review.openstack.org/287334 | 17:28 |
| mordred | clarkb: I think that should do it | 17:28 |
| *** kiran-r has quit IRC | 17:28 | |
| clarkb | mnaser: the file size seems to match. qcow2 is ok right? (it is what I uploaded manually yesterday and it booted so I think qcow2 is fine) | 17:29 |
| mnaser | yeah qcow2 is fine, this is plain kvm | 17:29 |
| mnaser | id just boot and check get-vnc-console and see what's there | 17:29 |
| *** sfinucan has quit IRC | 17:30 | |
| pabelanger | AJaeger: fungi: bindep for ansible-jobs look good | 17:30 |
| fungi | excellent | 17:30 |
| pabelanger | I even uploaded an empty other-requirements.txt and the job still passed | 17:31 |
| pabelanger | which is good, since ansible took care of the dependencies said job | 17:31 |
| fungi | pabelanger: AJaeger: i'm mostly hesitant to do much in the way of cutover to ubuntu-trusty workers this week for fear of further disrupting the already slammed feature-freeze-week gate load | 17:31 |
| *** dprince has quit IRC | 17:31 | |
| pabelanger | Ya, I didn't even think of that | 17:31 |
| fungi | if things start to die off friday that would be great timing, or else monday/tuesday next week | 17:32 |
| clarkb | mnaser: oh! I was looking at the wrong images, ya size is 0 on one I was trying to boot | 17:32 |
| pabelanger | fungi: AJaeger: sure WFM | 17:32 |
| mnaser | yep, probably a failed upload in that case | 17:32 |
| fungi | but i have hopes the gate will quiet down a little once the current impetus for the rush is over. trying not to introduce too many new fires to fight while we're still stretched thin keeping the existing situation stitched together | 17:32 |
| *** Swami has joined #openstack-infra | 17:33 | |
| *** vgridnev has joined #openstack-infra | 17:34 | |
| mnaser | how big are the images, i just tried to do an upload of a ~300M image from the office, just got uploaded with no problems, mordred earlier tried with manual shade upload and it went by fine? | 17:34 |
| anteaya | sanity speaks | 17:34 |
| clarkb | mordred: +2 on the logging change lets see if we can get that in | 17:34 |
| clarkb | mnaser: 5-6GB | 17:34 |
| clarkb | mnaser: they cache all of our git repos and the packages we use | 17:34 |
| mnaser | did mordred try to upload a 5-6gb image too? | 17:35 |
| *** mrmartin has quit IRC | 17:35 | |
| clarkb | mnaser: yes I think so | 17:35 |
| *** boris-42 has joined #openstack-infra | 17:36 | |
| *** jaypipes has joined #openstack-infra | 17:36 | |
| *** bpokorny has joined #openstack-infra | 17:36 | |
| mordred | mnaser: yes | 17:36 |
| mordred | I uploaded one of the nodepool ubuntu-trusty images from nodepool.o.o | 17:36 |
| mnaser | do uploads from nodepool happen concurrently | 17:37 |
| clarkb | mnaser: we do up to 4 at a time, but saw that it was failing even with a single upload happening | 17:37 |
| mnaser | i think i know what is going on. | 17:37 |
| *** hashar has quit IRC | 17:37 | |
| *** vivekd has quit IRC | 17:37 | |
| clarkb | mnaser: but yes when it started we attempted to do 4 at once | 17:37 |
| mnaser | nginx is buffering the request back to the slower glance backend | 17:37 |
| mnaser | so the file is uploaded really fast to nginx and glanceclient sits and wait for an answer | 17:37 |
| clarkb | aha | 17:37 |
| *** dmsimard|afk is now known as dmsimard | 17:38 | |
| mnaser | that takes a while, glanceclient gives up because nginx is still slowly feeding the data into glance | 17:38 |
| mnaser | if there was no buffering, this wouldn't happen because when the upload completes, it'll get a response right away | 17:38 |
| mnaser | based off this -- CentOS-6-x86_64-GenericCloud.qcow2c | 17:38 |
| mnaser | oops -- https://forum.nginx.org/read.php?2,213789,213793#msg-213793 | 17:38 |
| *** [1]Thelo has joined #openstack-infra | 17:38 | |
| mnaser | #justopenstackthings - i'll have a look and report back | 17:39 |
| clarkb | mnaser: do we need some sort of ping pong keepalive between nginx and glanceclient? | 17:39 |
| mnaser | clarkb: i think ideally nginx shouldn't buffer and if the communication betwen nginx and glance is slow, then glanceclient should suffer | 17:39 |
| clarkb | mnaser: that would work too :) thank you for looking into this | 17:39 |
| mnaser | the problem is the idea that the image was uploaded and shipped, but lack of response once the upload is done | 17:39 |
| *** sambetts is now known as sambetts|afk | 17:40 | |
| mnaser | i'd classify this as a problem at our side, probably doesnt happen when taking snapshots because that doesnt go through nginx | 17:40 |
| *** ccrouch1 has joined #openstack-infra | 17:40 | |
| clarkb | mnaser: do you think it is helpful to have nodepool keep trying or should we turn it off compltely for now while you work on your end? | 17:40 |
| mnaser | clarkb: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering | 17:41 |
| clarkb | we have ~152 imgaes we need to cleanup at this point | 17:41 |
| mnaser | i just have to flick this switch | 17:41 |
| *** Thelo has quit IRC | 17:41 | |
| *** [1]Thelo is now known as Thelo | 17:41 | |
| clarkb | woot | 17:41 |
| mnaser | actually | 17:41 |
| mnaser | http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_bufferingt | 17:41 |
| mnaser | http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering | 17:41 |
| mnaser | that | 17:41 |
| *** claudiub has quit IRC | 17:41 | |
| *** jordanP has quit IRC | 17:42 | |
| *** ccrouch has quit IRC | 17:42 | |
| *** yamamoto has joined #openstack-infra | 17:43 | |
| *** fhubik has quit IRC | 17:43 | |
| *** sdake has quit IRC | 17:43 | |
| *** vgridnev has quit IRC | 17:44 | |
| clarkb | I will work on cleaning up the images that we pushed that are all wonky | 17:44 |
| mnaser | clarkb: done, give it a shot, im off to go grab some nomnomz | 17:44 |
| clarkb | mnaser: thanks again! | 17:44 |
| mnaser | np, thank you! | 17:44 |
| *** vgridnev has joined #openstack-infra | 17:44 | |
| clarkb | (it may actually be a little while before nodepool gets to it as the other uploads to other clouds are happening too) | 17:45 |
| *** achanda has joined #openstack-infra | 17:45 | |
| *** jaypipes has quit IRC | 17:47 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 17:48 |
| *** dizquierdo has quit IRC | 17:49 | |
| *** dizquierdo has joined #openstack-infra | 17:50 | |
| *** yamamoto has quit IRC | 17:50 | |
| *** rockyg has quit IRC | 17:51 | |
| *** IlyaG has joined #openstack-infra | 17:52 | |
| *** rfolco has joined #openstack-infra | 17:52 | |
| *** electrofelix has joined #openstack-infra | 17:53 | |
| *** electrofelix has quit IRC | 17:53 | |
| *** _nadya_ has quit IRC | 17:53 | |
| *** asettle has quit IRC | 17:53 | |
| clarkb | ok image cleanup is in progress | 17:54 |
| clarkb | (to clean out the images that ended up in a funny state) | 17:54 |
| *** gongysh has quit IRC | 17:55 | |
| openstackgerrit | Sergey Kraynev proposed openstack/requirements: Bump upper-constraint for heatclient to 1.0.0 https://review.openstack.org/287351 | 17:55 |
| *** sputnik13 has joined #openstack-infra | 17:56 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Safely fail when logs cannot be found https://review.openstack.org/284999 | 17:56 |
| anteaya | AJaeger: in the next day or so I was hoping you might have a few moments to have a discussion with me about how to calculate how many of the jobs we are running (or have available to run) are non-voting as a percentage of total jobs we have available to run | 17:56 |
| *** Jeffrey4l has quit IRC | 17:56 | |
| anteaya | AJaeger: this came out of the discussion at the tc meeting yesterday | 17:57 |
| anteaya | AJaeger: I was thinking if we could grep the jjb xml somehow that would be a start | 17:57 |
| anteaya | AJaeger: I was wondering if you had any other or better thoughts | 17:57 |
| *** kencjohnston has quit IRC | 17:58 | |
| clarkb | and image cleanup is alldone | 17:58 |
| *** gokrokve has quit IRC | 17:58 | |
| clarkb | the next uploads should be starting from a clean slate and if this was the root cause end up working | 17:58 |
| *** kzaitsev_mb has quit IRC | 17:59 | |
| *** pcrews__ has joined #openstack-infra | 17:59 | |
| *** pcrews_ has quit IRC | 17:59 | |
| *** dprince has joined #openstack-infra | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| *** Sukhdev has quit IRC | 18:00 | |
| openstackgerrit | John Garbutt proposed openstack/requirements: Bump upper constraint on python-novaclient to 3.3.0 https://review.openstack.org/287355 | 18:00 |
| *** lucasagomes is now known as lucas-dinner | 18:01 | |
| *** jsavak has quit IRC | 18:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add missing gate-{name}-ansible-functional-ubuntu-trusty-nv https://review.openstack.org/287358 | 18:02 |
| *** jsavak has joined #openstack-infra | 18:03 | |
| *** maishsk has joined #openstack-infra | 18:03 | |
| jeblair | mordred, SpamapS: naturally, on my workstation, i need to set ANSIBLE_NOCOWS=1 | 18:03 |
| *** e0ne has joined #openstack-infra | 18:04 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 18:04 |
| mordred | jeblair: yes. it's an important env var to set | 18:04 |
| mordred | jeblair: or, you can just learn to love the cows | 18:04 |
| *** austin81 has quit IRC | 18:04 | |
| jeblair | mordred: i do love cows! | 18:04 |
| *** jsavak has quit IRC | 18:04 | |
| *** alivigni has quit IRC | 18:04 | |
| jeblair | which is why i have cowsay installed, unlike most servers, which is why i haven't run into this before :) | 18:04 |
| mordred | :) | 18:04 |
| *** jsavak has joined #openstack-infra | 18:04 | |
| *** shardy has quit IRC | 18:06 | |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-rackspace-experimental now exists with hot templating action | 18:08 |
| pabelanger | however the 'ready node' and 'error node' graphs look off | 18:08 |
| *** roxanaghe has quit IRC | 18:09 | |
| anteaya | sorry I missed the discussion about a rax experimental cloud/region | 18:10 |
| clarkb | jeblair: what do you think of spinning up more nodepool builder processes like we did in ft collins in order to get to the vexxhost uploads quicker | 18:10 |
| anteaya | what is the purpose of this region and why is it so named? | 18:10 |
| clarkb | jeblair: is there command history I can lookup for that? | 18:10 |
| fungi | anteaya: actually, the output from the zuul layout job lists every job it knows to run for all projects, and indicates which have a non-voting parameter too | 18:11 |
| fungi | anteaya: so it could probably be determined purely on analysis of that output | 18:11 |
| anteaya | oh okay grand | 18:11 |
| anteaya | that would be easy enough | 18:12 |
| *** keedya has joined #openstack-infra | 18:12 | |
| fungi | anteaya: based on my interpretation of your problem statement, i don't think you'd be able to figure out what you wanted from the jjb configs, since it's zuul which actually knows what votes and what doesn't | 18:12 |
| *** david-lyle has quit IRC | 18:12 | |
| mordred | yah - I also hadn't heard of rax experimental and am curious as to what it is | 18:12 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:12 | |
| anteaya | fungi: ah okay thank you | 18:12 |
| *** david-lyle has joined #openstack-infra | 18:13 | |
| jeblair | mordred, anteaya: see the discussion in here a while back. it's the *dashboard* that is experimental. | 18:13 |
| mordred | AH | 18:13 |
| anteaya | oh okay thank you | 18:13 |
| mordred | that makes much more sense to me | 18:13 |
| * anteaya goes to read backscroll | 18:13 | |
| *** fernnest has quit IRC | 18:13 | |
| jeblair | it's trying out the templating stuff | 18:13 |
| anteaya | ah | 18:13 |
| jeblair | clarkb: "nodepool-builder -d -c /etc/nodepool/nodepool.yaml" | 18:14 |
| jeblair | clarkb: but what we learned from that is --- | 18:14 |
| jeblair | clarkb: either use the existing logging config file or make a new one | 18:14 |
| clarkb | jeblair: oh right | 18:14 |
| jeblair | clarkb: (assuming you care about the output) | 18:14 |
| *** ihrachys has quit IRC | 18:14 | |
| clarkb | I can use tee to tee the stdout to files | 18:14 |
| clarkb | so that is easy enough | 18:14 |
| jeblair | that'll work too (though it may be stderr?) | 18:14 |
| clarkb | can catch both | 18:15 |
| *** sripriya_ has quit IRC | 18:15 | |
| *** dprince has quit IRC | 18:16 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Set the neutron parameters mandatory for controller https://review.openstack.org/284835 | 18:16 |
| *** dprince has joined #openstack-infra | 18:16 | |
| *** mikelk has quit IRC | 18:17 | |
| *** sridhar_ram has joined #openstack-infra | 18:17 | |
| *** mriedem has quit IRC | 18:18 | |
| clarkb | jeblair: and you ran that command as the nodepool user? | 18:18 |
| *** yaume_ has quit IRC | 18:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Add nodepool vexxhost dashboard for grafana https://review.openstack.org/287211 | 18:18 |
| jeblair | clarkb: yes | 18:18 |
| clarkb | cool vexxhost uploads are in progress now | 18:21 |
| *** angdraug has joined #openstack-infra | 18:21 | |
| *** tphummel has joined #openstack-infra | 18:22 | |
| cj | hey folks | 18:22 |
| cj | http://logs.openstack.org/56/287056/1/check/gate-gerritbot-pep8/d0c9194/ | 18:22 |
| cj | could someone help me find a pythontidy I can run my code through? | 18:22 |
| anteaya | what is a pythontidy? | 18:24 |
| *** annegent_ has joined #openstack-infra | 18:24 | |
| clarkb | anteaya: something to format the code in a way that is pep8 compliant | 18:24 |
| clarkb | I just use whatever python vim plugin ubuntu ships with | 18:24 |
| cj | I'm not sure if it exists. perltidy applies formatting rules to code, so I was hoping there was one for python | 18:25 |
| anteaya | ah | 18:25 |
| cj | clarkb: danke | 18:25 |
| anteaya | https://pypi.python.org/pypi/PythonTidy/ | 18:25 |
| *** abregman is now known as abregman|mtg | 18:25 | |
| anteaya | apparently there is | 18:25 |
| *** jsavak has quit IRC | 18:25 | |
| cj | okay, that's the one I found, but I can't seem to hook it in to emacs. | 18:25 |
| cj | clarkb: I'll open it with vim. is there a command I need to run to have it do its thing? | 18:26 |
| clarkb | cj: I am not sure how to get vim to apply the rules after the code is written | 18:26 |
| clarkb | I am sure there is a way but I always have it apply as I go | 18:27 |
| *** sarob has joined #openstack-infra | 18:27 | |
| cj | okay. | 18:27 |
| clarkb | oh if you emacs then jeblair or sdague may have a good asnwer for you | 18:27 |
| *** annegentle has quit IRC | 18:27 | |
| *** annegent_ has quit IRC | 18:28 | |
| *** austin81 has joined #openstack-infra | 18:28 | |
| *** dizquierdo has quit IRC | 18:29 | |
| lifeless | jeblair: mtreinish: the call is a flush call on a readonly file - I'd just remove the flush call as noise :) | 18:29 |
| *** mriedem has joined #openstack-infra | 18:29 | |
| *** e0ne has quit IRC | 18:29 | |
| *** ybathia has joined #openstack-infra | 18:29 | |
| mtreinish | lifeless: it already has been | 18:29 |
| *** dizquierdo has joined #openstack-infra | 18:29 | |
| *** e0ne has joined #openstack-infra | 18:30 | |
| mtreinish | lifeless: https://review.openstack.org/286928 | 18:30 |
| lifeless | mtreinish: ok cool; is there another on-closed error that that uncovered? | 18:30 |
| *** doug-fish has quit IRC | 18:30 | |
| *** jsavak has joined #openstack-infra | 18:30 | |
| *** doug-fish has joined #openstack-infra | 18:31 | |
| *** e0ne has quit IRC | 18:31 | |
| *** ddieterly has joined #openstack-infra | 18:31 | |
| mtreinish | lifeless: jeblair found the bug and fixed it with: https://review.openstack.org/287299 | 18:31 |
| *** e0ne has joined #openstack-infra | 18:32 | |
| clarkb | mnaser: OpenStackCloudException: Image creation failed: Error finding address for https://image-ca-ymq-1.vexxhost.net/v2/images/652eb40f-bde5-4e2b-adc2-8eb74b6d507f/file: Unable to establish connection to https://image-ca-ymq-1.vexxhost.net/v2/images/652eb40f-bde5-4e2b-adc2-8eb74b6d507f/file | 18:33 |
| openstackgerrit | C.J. Collier proposed openstack-infra/gerritbot: Improve gerritbot error handling https://review.openstack.org/287056 | 18:33 |
| clarkb | mnaser: that happened at 2016-03-02 18:30:32,944 UTC | 18:33 |
| clarkb | so about 3 minutes ago | 18:33 |
| pabelanger | jeblair: will have to dive into experimental grafana. If you select individual zones the graphs match up, when you select all for experimental, 2 don't sync. Maybe the template logic is not correct | 18:34 |
| pabelanger | either way, grafyaml did its job | 18:34 |
| anteaya | currently jeblair patch https://review.openstack.org/#/c/282099/ add timeout/retries to zuul cloner is in a cannot merge state | 18:34 |
| clarkb | mnaser: is it possible your change hasn't taken effect yet? | 18:34 |
| fungi | clarkb: "error finding address" seems specious. i wonder if that's a misleading error message | 18:34 |
| anteaya | jeblair: did you want to rebase? | 18:34 |
| *** doug-fis_ has joined #openstack-infra | 18:34 | |
| clarkb | fungi: it is I looked at glance client code and it looks like they copied the error message from the actual name lookup failure handling code | 18:34 |
| fungi | hah | 18:35 |
| fungi | classic | 18:35 |
| openstackgerrit | Austin Clark proposed openstack-infra/infra-specs: Deploy-Stackviz spec https://review.openstack.org/287373 | 18:35 |
| clarkb | fungi: the second part of that is the important bit "unable to establish connection to" is the actual python requests exception | 18:35 |
| *** doug-fish has quit IRC | 18:35 | |
| fungi | yeah, that seemed far more likely | 18:35 |
| clarkb | mnasers theory was caching in nginx confusing glanceclient while nginx finished writing to glance backend | 18:35 |
| AJaeger | anteaya: let me read backscroll, will have some time soonish | 18:36 |
| mnaser | clarkb: the changes should be in effect, unless http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering -- "When HTTP/1.1 chunked transfer encoding is used to send the original request body, the request body will be buffered regardless of the directive value unless HTTP/1.1 is enabled for proxying." | 18:36 |
| mnaser | we coud be fallin in that | 18:36 |
| austin81 | AJaeger: https://review.openstack.org/287373 :) | 18:37 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Switch to haproxy driver for heat lbaas v2 integration tests https://review.openstack.org/287377 | 18:37 |
| anteaya | AJaeger: thanks I think fungi suggested a scrape of the layout job output | 18:37 |
| *** abregman|mtg is now known as abregman|afk | 18:37 | |
| mnaser | ok so looks like glance does use chunked transfer so that directive is ignored | 18:37 |
| mnaser | i'll force the proxy to use http 1.1, which will make it effectie | 18:38 |
| anteaya | AJaeger: I'm still reading backscroll from last night so haven't gotten to that yet | 18:38 |
| *** fernnest has joined #openstack-infra | 18:38 | |
| *** doug-fis_ has quit IRC | 18:39 | |
| mnaser | clarkb: can you try again, i think the directive was ignored because it was a http/1.1 request | 18:39 |
| clarkb | mnaser: yup there are a couple other uploads queued | 18:39 |
| *** kushal has quit IRC | 18:40 | |
| mnaser | yep its proxying in http/1.1 -- lets see how this goes | 18:40 |
| *** piet has joined #openstack-infra | 18:40 | |
| *** infra-red has joined #openstack-infra | 18:41 | |
| *** sdake has joined #openstack-infra | 18:41 | |
| *** amitgandhinz has quit IRC | 18:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 18:43 |
| *** amitgandhinz has joined #openstack-infra | 18:45 | |
| *** ajmiller_ has joined #openstack-infra | 18:45 | |
| openstackgerrit | Merged openstack-infra/shade: Add debug message about file hash calculation https://review.openstack.org/287281 | 18:45 |
| *** ajmiller has quit IRC | 18:48 | |
| openstackgerrit | Merged openstack/requirements: Cap osprofiler https://review.openstack.org/278984 | 18:48 |
| AJaeger | fungi, understood for ubuntu-trusty cutover. What do you think of reviewing 285722 and 283445 now - so thta we can make changes if needed before? 285722 looks straightforward to me but on 283445 I'd like you to check the tox change | 18:49 |
| openstackgerrit | Merged openstack/requirements: Bump python-ironic-inspector-client to 1.5.0 https://review.openstack.org/285972 | 18:49 |
| *** austin81 has left #openstack-infra | 18:50 | |
| *** jcoufal has quit IRC | 18:50 | |
| *** esikache1 has joined #openstack-infra | 18:51 | |
| *** pvaneck has joined #openstack-infra | 18:53 | |
| AJaeger | anteaya: yeah, he's suggestion is what I would have suggested as well. Run "tox -e zuul" and take it from there. | 18:54 |
| *** maishsk has quit IRC | 18:54 | |
| *** sripriya has joined #openstack-infra | 18:54 | |
| sdague | cj: https://dague.net/2015/03/24/openstack-emacs-tools/ | 18:54 |
| *** vgridnev has quit IRC | 18:55 | |
| anteaya | AJaeger: sounds good, thanks | 18:55 |
| anteaya | glad it is so simple | 18:55 |
| *** esikache1 has quit IRC | 18:56 | |
| clarkb | mnaser: woot I think we just had an image go ready in vexxhost | 18:56 |
| *** jcoufal has joined #openstack-infra | 18:56 | |
| clarkb | mnaser: 7ca18474-3b2e-4cff-91a0-0ea34e722e8b is the image uuid | 18:57 |
| mnaser | wewt | 18:57 |
| fungi | AJaeger: seems fine to me. ideally the database-using tox jobs would become the exception, but i don't know of an easy way to work out which projects/jobs are relying on mysql or postgres databases set up in advance | 18:57 |
| clarkb | mnaser: file size looks good too | 18:57 |
| clarkb | may be time to revert the max-servers: -1 change | 18:57 |
| mnaser | yeah i think its resolved, as explained by the much slower time it showed in the logs :-P | 18:57 |
| clarkb | I will try booting off that image real quick now | 18:58 |
| anteaya | fungi: AJaeger I think dhellmann was looking for a ballpark percentage | 18:58 |
| *** sripriya_ has joined #openstack-infra | 18:58 | |
| *** kushal has joined #openstack-infra | 18:58 | |
| anteaya | so hopefully grepping the output of tox -e zuul will get us a starting point | 18:58 |
| fungi | anteaya: my ballpark percentage is "no clue" | 18:58 |
| anteaya | yup, that is where I started too | 18:58 |
| fungi | so, yeah, need to find actual numbers | 18:58 |
| *** maishsk has joined #openstack-infra | 19:00 | |
| *** thorst_ has quit IRC | 19:00 | |
| *** thorst has joined #openstack-infra | 19:01 | |
| *** thorst has quit IRC | 19:01 | |
| *** thorst has joined #openstack-infra | 19:01 | |
| AJaeger | fungi, if database-using-tox should be the exception, should we use different naming? Create also a tox 'gate-{name}-tox-db-{envlist}' and change my 'gate-{name}-tox-nodb-{envlist}' to something else? Or create both versions? | 19:01 |
| *** sripriya has quit IRC | 19:01 | |
| fungi | AJaeger: i don't really have a strong opinion on it. what you have now looks fine honestly | 19:02 |
| AJaeger | glad to hear ;) | 19:03 |
| clarkb | mnaser: I successfully booted off of that image so looks good | 19:03 |
| clarkb | ssh'd in and everything | 19:03 |
| anteaya | yay | 19:04 |
| *** jsavak has quit IRC | 19:04 | |
| mnaser | \o/ | 19:04 |
| mnaser | thanks for helping us find out a problem already i guess :-P | 19:04 |
| fungi | one of the fringe benefits of donating ;) | 19:05 |
| *** annegentle has joined #openstack-infra | 19:05 | |
| AJaeger | fungi, https://etherpad.openstack.org/p/bindep contains all info about the ubuntu-trusty change... | 19:05 |
| *** jsavak has joined #openstack-infra | 19:06 | |
| clarkb | hrm | 19:07 |
| clarkb | our ubuntu-trusty images are more than twice the size of our devstack-trusty images | 19:07 |
| clarkb | or thereabouts twice as big | 19:08 |
| fungi | that seems odd | 19:08 |
| clarkb | so much for minimal :) | 19:08 |
| fungi | pre-caching more packages? | 19:08 |
| clarkb | 5.7GB vs 11GB | 19:08 |
| clarkb | fungi: maybe? I havne't dug further than looking at ls output funny | 19:08 |
| fungi | still, 4.3gb of packages is... a lot | 19:08 |
| fungi | so i doubt that's it | 19:08 |
| pleia2 | that is a lot, are they shipping a desktop? :) | 19:09 |
| clarkb | they should have almost nohting in them because the are "minimal" | 19:09 |
| pleia2 | I just installed 14.04.4 desktop yesterday and even that only takes up 8.6G | 19:10 |
| mordred | clarkb: we should definitely figure that out - they should certainly have less | 19:10 |
| mnaser | what does df -h say on these images | 19:10 |
| bkero | That's a crazy amount of space for an os image | 19:10 |
| mnaser | once they boot | 19:10 |
| *** daemontool_ has quit IRC | 19:10 | |
| fungi | perhaps we aren't properly squashing them? | 19:10 |
| fungi | yeah, i'm thinking what mnaser is probably also getting at | 19:11 |
| mnaser | ^ my thoughts, virt-sparsify | 19:11 |
| pleia2 | and that's 8.6G on the filesystem, while it's running | 19:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Revert "Disable image uplaods to vexxhost" https://review.openstack.org/287389 | 19:11 |
| clarkb | bkero: well we cache stuff | 19:11 |
| AJaeger | armax, dougwig, mestery : https://review.openstack.org/#/c/282572/ needs one neutron liason review, could either of you look at it, please? | 19:11 |
| clarkb | bkero: but ~5GB is about what we expect | 19:11 |
| bkero | clarkb: Yeah. I'm hearing 4.3GB of packages, so 700mb for an OS makes sense | 19:11 |
| mestery | AJaeger: Looking | 19:11 |
| bkero | 6GB doesn't make sense | 19:11 |
| clarkb | fungi: mordred pleia2 https://review.openstack.org/287389 will keep us moving forward on vexxhost | 19:12 |
| pleia2 | clarkb: yay | 19:12 |
| *** doug-fish has joined #openstack-infra | 19:12 | |
| fungi | bkero: a lot of it is the full mirror of all our git repos we keep locally on images, but that should be on both so doesn't account for the discrepancy | 19:12 |
| *** kencjohnston has joined #openstack-infra | 19:12 | |
| dougwig | AJaeger: there you go, all three. | 19:13 |
| clarkb | /dev/disk/by-label/cloudimg-rootfs 188G 10G 170G 6% / | 19:13 |
| clarkb | so we are using roughly that much disk | 19:13 |
| clarkb | I think | 19:13 |
| fungi | curious to know where the bulk of that is residing in the fs | 19:13 |
| clarkb | ya I am running the very expensive du -hs * in / | 19:14 |
| bkero | clarkb: pastebin? | 19:14 |
| clarkb | bkero: its not done yet | 19:14 |
| bkero | I mean when it is :) | 19:14 |
| clarkb | sure | 19:14 |
| bkero | baobab is a really cool tool for doing that, but would probably be hellish over ssh | 19:15 |
| AJaeger | dougwig, mestery, armax : Thanks! | 19:15 |
| *** ajmiller_ is now known as ajmiller | 19:15 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 19:16 |
| clarkb | http://paste.openstack.org/show/489033/ that implies it is our cache, will dig in /opt more | 19:16 |
| fungi | yeah, i think the only thing we cache in /opt is git repos for the moment | 19:16 |
| clarkb | ya 6.8GB is just /opt/git | 19:17 |
| fungi | so this may be because we | 19:17 |
| pleia2 | wow :\ | 19:17 |
| fungi | grr | 19:17 |
| clarkb | fungi: hrm? | 19:17 |
| fungi | stray finger on the enter key | 19:17 |
| fungi | we're trying to cache all branches of all repos rather than just the master branch state (which was our previous naive use of the dib source element) | 19:18 |
| mnaser | time to run reverse cache proxies on mirror nodes and use that to pull deps? | 19:18 |
| clarkb | fungi: but devstack-trusty uses the same caching thing | 19:18 |
| clarkb | mnaser: well if we can get it back down to 5.7GB I might be happy with that since we essentially have the same images | 19:18 |
| fungi | double-checking how we're calling it from the ubuntu-trusty config | 19:19 |
| sdague | hey, remember that 180 minute backstop idea? | 19:19 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Don't extend the passed in targets https://review.openstack.org/287299 | 19:19 |
| *** kushal has quit IRC | 19:19 | |
| *** korzen has quit IRC | 19:19 | |
| openstackgerrit | devdatta-kulkarni proposed openstack/requirements: Adding python-solumclient https://review.openstack.org/287313 | 19:20 |
| sdague | we should consider that, because especially on environments like internap we can become our own noisy neighbors when at full utilization | 19:20 |
| fungi | clarkb: looking at the diskimages list, the only differences there shouldn't impact git caching, i agree | 19:20 |
| sdague | and drive everything past 2 huors | 19:20 |
| sdague | https://review.openstack.org/#/c/286276/ | 19:20 |
| *** abitha has joined #openstack-infra | 19:20 | |
| mnaser | our glance images are currently stored in a single az (triple replicated on ceph). we're looking at deploying glance-api with cache on the other az to make things faster | 19:20 |
| sdague | 2 fails in the gate, on internap, because the job was still going at 2 hours | 19:20 |
| sdague | but it was passing | 19:21 |
| mnaser | so if that becomes an annoyance, we might do that at our side | 19:21 |
| *** Sukhdev has joined #openstack-infra | 19:21 | |
| bkero | clarkb: are the repos copied out of /opt/git when they're used? | 19:21 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:22 | |
| bkero | or are the 'git checkout $THING' in place? | 19:22 |
| bkero | they* | 19:22 |
| fungi | clarkb: i see we're not doing QEMU_IMG_OPTIONS: compat=0.10 on ubuntu-trusty... should we be? | 19:22 |
| *** Sukhdev has quit IRC | 19:22 | |
| clarkb | fungi: I think hpcloud was the only cloud that required it but it could be a difference in compression between versions so maybe we should | 19:22 |
| clarkb | bkero: they use gits hardlinking cloning stuff iirc | 19:23 |
| *** infra-re_ has joined #openstack-infra | 19:23 | |
| bkero | Squashfs the repos? :3 | 19:23 |
| openstackgerrit | Merged openstack-infra/project-config: networking-odl: enable test with ODL release of boron-snapshot https://review.openstack.org/282572 | 19:23 |
| *** alivigni has joined #openstack-infra | 19:23 | |
| *** jed56 has quit IRC | 19:23 | |
| *** abitha has left #openstack-infra | 19:23 | |
| *** infra-red has quit IRC | 19:25 | |
| *** infra-red has joined #openstack-infra | 19:26 | |
| fungi | sdague: i wonder if that means we're overutilizing/oversubscribed too far and need a lower quota to stop adversely impacting their systems, or if there might be something else going on in internap | 19:26 |
| mgagne | fungi: currently running at 2:1 (instead of 3:1 a couple of months ago) | 19:27 |
| sdague | fungi: right, I don't know, but moving that backstop up to 180minutes would at least not fail good jobs | 19:27 |
| fungi | sdague: the jobs just ran slower, no indication they spent a lot of time updating git repos or anything where they might have been hung on a git remote action? | 19:27 |
| *** maishsk has quit IRC | 19:27 | |
| sdague | fungi: yeh, no specific hot spot | 19:27 |
| sdague | just everything is going slow | 19:27 |
| *** ociuhandu has quit IRC | 19:27 | |
| sdague | pip_install is like 2x what I'd expect | 19:28 |
| sdague | my guess is disk io contention | 19:28 |
| clarkb | fungi: when I can get out of my chair and boot my test machine I will run some local dib build tests to see if the compat option is making things saner | 19:28 |
| fungi | i suppose increasing the timeout is an option. i don't know for sure whether that removes some of our only backpressure on people merging things which will make the jobs run longer and soon start creeping up to the new timeout | 19:28 |
| sdague | I don't think that timeout is really back preasure on that | 19:28 |
| sdague | we've got social preasure well before that | 19:29 |
| *** infra-re_ has quit IRC | 19:29 | |
| sdague | I actually think a high percentage of our current gate reset is basically a combo of 2 timeouts right now | 19:30 |
| sdague | that we are triggering because of load on providers | 19:30 |
| sdague | which... means rechecks, generating more load | 19:30 |
| sdague | so vicious cycle | 19:30 |
| fungi | i'm not particularly opposed, though i would love to get a better handle on whether we're seeing new performance regressions in particular providers so either we or they can address that | 19:30 |
| anteaya | dhellmann AJaeger fungi: http://paste.openstack.org/show/489038/ 5.87% of total jobs that zuul knows about are non-voting | 19:30 |
| *** caphrim007 has joined #openstack-infra | 19:30 | |
| sdague | sure, once openstack health is back that has a set of data | 19:30 |
| anteaya | according to my grepping + math | 19:30 |
| sdague | but we're blind there atm | 19:30 |
| fungi | right | 19:30 |
| AJaeger | anteaya: only? Wow | 19:31 |
| sdague | honestly, the timeout reset is super hard to spot like this unless you have a patch impacted by it | 19:31 |
| anteaya | AJaeger: yeah I thought it would be higher too | 19:31 |
| sdague | because there is nothing dead on the zuul queue to look at | 19:32 |
| AJaeger | anteaya: can you calculate also jobs run in check queue only | 19:32 |
| anteaya | I'll see | 19:32 |
| sdague | I happened to have a patch in gate that got hit by it twice | 19:32 |
| clarkb | when jobs timeout they should report failure | 19:32 |
| clarkb | so you would see fail by timeout | 19:32 |
| anteaya | AJaeger: not sure how since they are all called gate | 19:32 |
| fungi | anteaya: the output zuul produces is per pipeline | 19:33 |
| fungi | anteaya: so you could trim it down to just the output for one pipeline | 19:33 |
| anteaya | fungi: ah okay I will continute to browse the file and see if I can get a number | 19:34 |
| *** doug-fis_ has joined #openstack-infra | 19:35 | |
| caphrim007 | hey folks, i'm having trouble getting shade to list my servers. http://pastebin.com/HPrHSA0d is the error i'm receiving. its a packstack install | 19:35 |
| *** Sukhdev has joined #openstack-infra | 19:35 | |
| AJaeger | anteaya: I just manually edit the file and copied the check pipelines out. | 19:35 |
| AJaeger | anteaya: I have a different tree then you have, so numbers are "ball-park" figures only | 19:35 |
| AJaeger | 586 non-voting out of 4101 check jobs | 19:36 |
| *** doug-fish has quit IRC | 19:36 | |
| fungi | though also, this number is likely only useful for limited purposes. the frequency with which these jobs are run is unrelated to the number of them which exist | 19:37 |
| anteaya | AJaeger: fungi I've updated the paste look at line 10 | 19:37 |
| anteaya | I might not be grepping properly | 19:37 |
| *** rockyg has joined #openstack-infra | 19:37 | |
| *** exploreshaifali has quit IRC | 19:38 | |
| fungi | you could probably get that out of graphite or logstash, but then even frequency with which they're run is not a good indicator of resources consumed without taking into account how long each runs | 19:38 |
| anteaya | AJaeger: I don't understand how you are getting those numbers, but I'm not having confidence in my method to get accurate numbers | 19:38 |
| AJaeger | anteaya: your grep looks fine, just wanted to say that I'm not on head | 19:38 |
| anteaya | AJaeger: oh | 19:38 |
| fungi | if the goal is figuring out how much of our worker pool gets spent running non-voting jobs, the figure is probably going to not be easy to arrive at | 19:39 |
| anteaya | fungi: very true, these are just the jobs that exist | 19:39 |
| AJaeger | anteaya: I had 12422 and 689 jobs overall... | 19:39 |
| anteaya | fungi: that seems to be the conclusion I am reaching as well | 19:39 |
| *** ybathia has quit IRC | 19:39 | |
| anteaya | AJaeger: 12422 what? | 19:39 |
| *** davideagnello is now known as davideagnello__ | 19:40 | |
| AJaeger | 12422 total jobs - and 689 non-voting. So same ballpark figure as yours... | 19:40 |
| anteaya | oh okay thanks | 19:40 |
| anteaya | so a very rough ballpark | 19:41 |
| *** infra-re_ has joined #openstack-infra | 19:41 | |
| anteaya | which likely isn't of any value in helping us decide how to best use resources | 19:41 |
| anteaya | dhellmann: ^^ | 19:41 |
| *** korzen has joined #openstack-infra | 19:41 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: mostly standardize dsvm tempest jobs on 180 minute backstop https://review.openstack.org/287407 | 19:43 |
| sdague | fungi ^^^ | 19:43 |
| pabelanger | when we send out ATC passes for the summit, do we check against the current development cycle or the previous? So with Austin, you need commits in liberty or mitaka? | 19:44 |
| *** infra-red has quit IRC | 19:44 | |
| clarkb | pabelanger: its the current cycle | 19:44 |
| clarkb | so mitaka | 19:44 |
| clarkb | pabelanger: in the past it was the current and previous but that has been adjusted based on scale? | 19:44 |
| fungi | we could probably, with some effort, determine aggregate runtime for each job (as a product of run count and run duration) from graphite, then map that to the information zuul has about whether each is voting or non-voting since that's determined purely on job name | 19:44 |
| pabelanger | clarkb: right, so how does commits after the ATC email work? | 19:44 |
| anteaya | fungi: interesting | 19:45 |
| *** davideagnello__ is now known as davide85_ | 19:45 | |
| *** Kaiyan has joined #openstack-infra | 19:45 | |
| clarkb | pabelanger: there are several discount pass waves sent out and they stop before the expiration on the early reg iirc | 19:46 |
| clarkb | pabelanger: so you hvae to get in befor ethat | 19:46 |
| *** doug-fish has joined #openstack-infra | 19:46 | |
| fungi | pabelanger: criteria are decided by the event organizers, but this time around it's basically anyone who owns a change merged to a tc-recognized project deliverable between the last release and the integrated feature freeze | 19:46 |
| *** Sukhdev has quit IRC | 19:46 | |
| pabelanger | fungi: clarkb: perfect | 19:46 |
| *** korzen has quit IRC | 19:46 | |
| fungi | so october 15 through probably march 3 this time | 19:46 |
| fungi | as my plan is to try and fire off the final batch tomorrow | 19:47 |
| *** doug-fis_ has quit IRC | 19:47 | |
| *** rguillebert has quit IRC | 19:47 | |
| openstackgerrit | Merged openstack-infra/shade: Add test for os_server Ansible module https://review.openstack.org/285424 | 19:48 |
| pabelanger | fungi: do we have the documented some place on the wiki? | 19:49 |
| fungi | pabelanger: no, since it varies from cycle to cycle it's generally not been well tracked | 19:49 |
| pabelanger | okay, thanks | 19:49 |
| *** infra-red has joined #openstack-infra | 19:50 | |
| fungi | also i feel like publishing explicit qualifying requirements increases the number of people who will abuse it for free conference admission, as opposed to a more nebulous "we give free passes to active contributors for the cycle" | 19:50 |
| *** infra-re_ has quit IRC | 19:52 | |
| fungi | for the most part, people actually working on upstream development in official projects should never need to ask. most of the questions i get about it are people who are trying to get one commit in at the last minute so they don't have to pay $600, but aren't actually planning to participate in the design summit at all and just want to go to the conference for free | 19:53 |
| *** tongli has joined #openstack-infra | 19:53 | |
| *** jamesmcarthur has quit IRC | 19:54 | |
| anteaya | there are also the extra-atcs category for anyone who makes a contribution that gerrit doesn't know about | 19:55 |
| fungi | i get tons of questions like "here's a one-line patch changing a couple words in a code comment which i and several of my colleagues are listed as co-authors on as our only contribution, where's our free pass?" (how does a change like that even get approved by core reviewers on a project?), or "i had a patch merged in january of last year but haven't done anything since then, can i still get a free | 19:56 |
| fungi | conference pass?" | 19:56 |
| mordred | I should not respond to mailng lists | 19:56 |
| fungi | oh, and several this time were "i got a free pass for tokyo but couldn't attend, can i use it to get into the conference in austin?" | 19:56 |
| *** kzaitsev_mb has joined #openstack-infra | 19:57 | |
| Kaiyan | https://review.openstack.org/#/c/286179/ Hello everyone! Can anyone look at this change and approve it please? Thanks! | 19:57 |
| anteaya | fungi: yeah it is the addition of a whitespace or newline with 3 co-authors that boggles my mind | 19:58 |
| Clint | you should answer that you hid the golden tickets in a chocolate bar | 19:58 |
| *** sdake_ has joined #openstack-infra | 19:59 | |
| fungi | Clint: oh, i've got a golden ticket! | 19:59 |
| fungi | openstack conference wonkypass | 19:59 |
| *** BobBall is now known as BobBall_AWOL | 20:00 | |
| *** BobBall_AWOL has quit IRC | 20:00 | |
| *** sdake has quit IRC | 20:00 | |
| *** jamesmcarthur has joined #openstack-infra | 20:00 | |
| fungi | sdague: at what point did we start being able to infer the devstack-gate inner timeout from the jenkins timeout wrapper value? | 20:00 |
| *** tqtran has joined #openstack-infra | 20:00 | |
| *** thorst has quit IRC | 20:01 | |
| AJaeger | Kaiyan: not everybody here reviews everything. Please tell us a bit more about your change so that the right people can look at it - if they have time... | 20:01 |
| *** maishsk has joined #openstack-infra | 20:01 | |
| *** thorst has joined #openstack-infra | 20:01 | |
| AJaeger | fungi: 3rd of February jeblair created a patch - see Iff5e96bd80b8f165ccc7d6be09b53208613571eb | 20:01 |
| fungi | sdague: oh, i bet it defaults to 120 | 20:01 |
| fungi | AJaeger: ooh! | 20:02 |
| openstackgerrit | Merged openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 20:02 |
| morgan | mordred: lol | 20:02 |
| * mordred goes back to hiding | 20:02 | |
| fungi | AJaeger: sdague: that's some serious rainbow unicorn magic. i'm thrilled | 20:02 |
| *** sdake has joined #openstack-infra | 20:03 | |
| fungi | jeblair: ^ sorry i missed 275908, that's awesome | 20:03 |
| *** piet has quit IRC | 20:03 | |
| mtreinish | jeblair: https://pypi.python.org/pypi/subunit2sql/1.4.0 now to wait for puppet on subunit-worker01.o.o | 20:03 |
| AJaeger | fungi: and then we removed the TIMEOUT variable everywhere in project-config... | 20:04 |
| fungi | AJaeger: yep, i see that | 20:05 |
| fungi | i simply missed introduction of the magic | 20:05 |
| AJaeger | ;) | 20:05 |
| *** thorst has quit IRC | 20:05 | |
| *** sdake_ has quit IRC | 20:06 | |
| *** hashar has joined #openstack-infra | 20:06 | |
| *** thorst has joined #openstack-infra | 20:08 | |
| jeblair | mordred: that post is far more sensible than i was expecting based on your earlier warning :) | 20:10 |
| mordred | jeblair: it's all about setting expectatoins | 20:10 |
| jeblair | fungi: i think zaro did the hard work adding the feature to jenkins | 20:10 |
| Kaiyan | https://review.openstack.org/#/c/286179/ This review is to set monasca persister to only run monasca tempest tests in check and gate jobs instead of running all the 1000+ tests. Please review it when you get a chance! Thank you! | 20:10 |
| mordred | I successfully set your expectations low enough that the post seemed sensible | 20:10 |
| mordred | compared to the horror-show you know it COULD have been :) | 20:11 |
| fungi | jeblair: oh, yes i do remember him working on the jenkins side of that--thanks zaro! it's a massive simplification for our job configs and has been a long time coming | 20:11 |
| ianw | mordred: ahh, i did think it seemed like we should pass username/password in when i was going through the swift upload (https://review.openstack.org/#/c/287224/). so do we think it's fixed? | 20:11 |
| mordred | ianw: yah - we did the same thing in occ for normal swift Connection | 20:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ZUUL_REFNAME (1/2) for branch checking https://review.openstack.org/287414 | 20:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ZUUL_REFNAME (2/2) for branch checking https://review.openstack.org/287415 | 20:12 |
| AJaeger | fungi, here's the REFNAME cleanup we discussed earlier ^ | 20:12 |
| mordred | ianw: so I think between your patch and that patch we should be much better now | 20:12 |
| mordred | I need to cut a new shade release, but there was enough going on that I thought maybe today was a bad time? | 20:13 |
| *** infra-red has quit IRC | 20:13 | |
| openstackgerrit | gordon chung proposed openstack/requirements: bump ceilometermiddleware upper constraints https://review.openstack.org/287416 | 20:13 |
| mordred | infra-root: ^^ any thoughts on fear/joy related to a new shade release? | 20:13 |
| ianw | mordred: i haven't had my first coffee yet, let alone read scrollback :) | 20:13 |
| mordred | ianw: haha | 20:14 |
| mordred | ianw: DRINK MOAR COFFEEEE | 20:14 |
| ianw | probably the world is on fire | 20:14 |
| openstackgerrit | Douglas Mendizábal proposed openstack/requirements: Update upper-constraints for barbianclient https://review.openstack.org/287417 | 20:14 |
| anteaya | mordred: I have no feelings on the new shade release, so I guess I don't have any fear | 20:15 |
| anteaya | joy in as much as it has the potential to make others happy | 20:15 |
| *** claudiub has joined #openstack-infra | 20:15 | |
| * anteaya goes to find food | 20:15 | |
| mgagne | found a cool online dstat graphing tool: http://lamada.eu/dstat-graph/ | 20:16 |
| *** jsavak has quit IRC | 20:16 | |
| mgagne | can also be downloaded from https://github.com/Dabz/dstat_graph | 20:16 |
| *** bhunter_ has joined #openstack-infra | 20:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a third scenario for packstack integration testing https://review.openstack.org/286579 | 20:16 |
| *** piet has joined #openstack-infra | 20:17 | |
| *** doug-fis_ has joined #openstack-infra | 20:17 | |
| *** claudiub has quit IRC | 20:17 | |
| cj | I've fixed my gerrit commit. Is there a way to tell jenkins to kick off another test run? | 20:17 |
| *** sripriya__ has joined #openstack-infra | 20:17 | |
| cj | https://review.openstack.org/#/c/287056/ | 20:17 |
| *** claudiub has joined #openstack-infra | 20:17 | |
| *** doug-fish has quit IRC | 20:18 | |
| *** keedya has quit IRC | 20:19 | |
| cj | ah. To re-run check or gate jobs, leave a comment on the review with the form “recheck”. | 20:19 |
| *** maishsk_ has joined #openstack-infra | 20:20 | |
| *** jsavak has joined #openstack-infra | 20:20 | |
| *** maishsk has quit IRC | 20:20 | |
| *** maishsk_ is now known as maishsk | 20:20 | |
| AJaeger | cj: we automatically run a new check when you upload a new change | 20:20 |
| *** sripriya_ has quit IRC | 20:20 | |
| *** reed_ has joined #openstack-infra | 20:21 | |
| anteaya | cj: go to status.openstack.org/zuul and put 287056 in the Filters: text box | 20:22 |
| anteaya | cj: your patch is in the check queue awaiting a node | 20:22 |
| anteaya | and has been for an hour and 49 minutes | 20:23 |
| fungi | mordred: holding off non-urgent behavior changes to our primary ci toolchain makes sense. i think the devs currently bum-rushing feature freeze appreciate our change slushiness this week | 20:23 |
| *** annegentle has quit IRC | 20:23 | |
| cj | cool, thanks. I'd like to merge in changes from 279169 - is there an easy command to do that? | 20:23 |
| *** apoorvad has quit IRC | 20:23 | |
| mordred | fungi: yah. that's kinda what I was thinking | 20:24 |
| mordred | fungi: wait - maybe we've been taking the wrong approach this entire time | 20:24 |
| *** annegentle has joined #openstack-infra | 20:24 | |
| anteaya | cj: yes in your workspace go to the master branch with no changes going with you, then git review -d 279169 | 20:24 |
| mordred | fungi: maybe we should ONLY make changes to the infrastructure during feature freeze rush | 20:24 |
| anteaya | which creates a new branch with Gabor's change on it | 20:24 |
| mordred | fungi: to possibly block some percentage of the last-minute bad ideas | 20:25 |
| rcarrillocruz | haha | 20:25 |
| anteaya | then put your change as a child of 279169 | 20:25 |
| fungi | mordred: you can never block enough of the bad ideas to make that worthwhile | 20:25 |
| fungi | there are simply too many | 20:25 |
| mordred | fungi: good point | 20:25 |
| * mordred glares at floating ips | 20:25 | |
| *** ccrouch1 has quit IRC | 20:25 | |
| cj | anteaya: and if I wanted to merge these changes in to my branch? | 20:26 |
| mordred | fungi: hey, wanna buy a server that's only available through a NAT connection? I'll sell you one for $5 ... | 20:26 |
| mordred | cj: you do not want to do that | 20:26 |
| cj | same but in my branch instead of master? | 20:26 |
| anteaya | cj: we don't merge locally | 20:26 |
| cj | mordred: oh, alright. How about if I want to put my branch in a state that it can merge gracefully? | 20:26 |
| fungi | mordred: i can give you a really cheap rate on servers you'll never be able to reach | 20:26 |
| anteaya | we create individual patches and zuul merges | 20:26 |
| mordred | cj: yes, that is a thing you can enjoy doing | 20:26 |
| cj | mordred: heh, okay. I meant how do I find the merge conflicts? | 20:26 |
| mordred | cj: so - git review -d 279169 will make a local branch for you and download the change in question into that branch | 20:26 |
| mordred | cj: if you then cherry-pick your change on top of it | 20:27 |
| cj | okay, thanks. | 20:27 |
| mordred | cj: you can resolve the merge conflicts | 20:27 |
| * anteaya wonders what the difference is between what mordred and what she had said | 20:27 | |
| mordred | cj: and when you run git review again, it will upload your change, except now it will be parented on the 279169 change | 20:27 |
| mordred | anteaya: you said the same words - but you said less of them | 20:27 |
| ianw | fungi / jeblair : when you have a moment -> https://review.openstack.org/#/c/285876/ ; fixing up fight between pip/packages on fedora-minimal images. this feels like the right place to do it, because puppet shouldn't know about it, and without we leave the system in a really precarious state | 20:28 |
| anteaya | ah more words were needed | 20:28 |
| AJaeger | sdague: https://review.openstack.org/#/c/286085 is a nova vif change for project-config. Do you want t o review that? | 20:28 |
| *** andreykurilin__ has joined #openstack-infra | 20:28 | |
| mordred | anteaya: I'm assuming that cj is used to a different way of doing git and that all of our things are strange and frightening, so trying to step back and remember which things of how things work here he might not assume | 20:29 |
| anteaya | I'm not often accused of not using enough words | 20:29 |
| cj | furrfu. looks like PythonTidy does not emit code that passes pep8 | 20:29 |
| mordred | anteaya: I'm NEVER accused of that | 20:29 |
| anteaya | yup | 20:29 |
| mordred | :) | 20:29 |
| anteaya | he he | 20:29 |
| *** matrohon has joined #openstack-infra | 20:29 | |
| cj | thanks to both of you, mordred and anteaya | 20:29 |
| mordred | cj: I have thus far found no programs that are good at reformatting python code | 20:29 |
| mordred | cj: I've looked a few times | 20:29 |
| *** rguillebert has joined #openstack-infra | 20:29 | |
| cj | mordred: that's too bad. I like perltidy. I wonder if I can alter it to work with python, ruby, javascript, etc. | 20:30 |
| mordred | cj: well - you might be better off trying to morph https://pypi.python.org/pypi/pyfmt/ into shape | 20:30 |
| *** keedya has joined #openstack-infra | 20:31 | |
| *** annegentle has quit IRC | 20:31 | |
| fungi | mordred: closest i've found is autopep8. it's not perfect but it's not terrible for trying to neaten up old projects which didn't follow any consistent style | 20:31 |
| mordred | the biggest problem I have with pyfmt is that it disagrees with me on how to deal with line continuations after parentheses | 20:31 |
| *** doug-fish has joined #openstack-infra | 20:31 | |
| mordred | _I_ prefer to line break after opening parens a-la lisp | 20:31 |
| *** ccrouch has joined #openstack-infra | 20:31 | |
| mordred | and tryto avoid visual indentation blocks at all costs | 20:31 |
| fungi | https://pypi.python.org/pypi/autopep8 | 20:31 |
| mordred | pyfmt thinks the opposite is what should happen | 20:31 |
| fungi | i still end up cleaning up behind it and turning off some rules, but it's a start | 20:32 |
| *** jsavak has quit IRC | 20:32 | |
| fungi | and faster than hand-formatting the entire project in many cases | 20:32 |
| AJaeger | any project-config core around to +2 a change to fix the bug association for training-guides, please? https://review.openstack.org/#/c/283493/ | 20:32 |
| mordred | fungi: I should see if I can get autopep8 to line-break the way I like | 20:32 |
| *** jsavak has joined #openstack-infra | 20:33 | |
| *** doug-fis_ has quit IRC | 20:33 | |
| AJaeger | jeblair: do you want to merge the wheel mutex change now? Or wait until M3 is over? https://review.openstack.org/275975 | 20:33 |
| fungi | mordred: if memory serves (been months since the last time i used it) i turned off line breaking in it and used a regex to find long lines so i could fix them myself | 20:33 |
| mordred | that's probably the best idea | 20:33 |
| fungi | or to find lines i'd long ago used backslash continuations on and have since thought better of | 20:33 |
| AJaeger | One more change for wrong bug association: https://review.openstack.org/#/c/283453/ | 20:34 |
| mordred | my current rubric is "if I can't figure out how to paren-break a line, I probably need to make a helper method or function" :) | 20:34 |
| AJaeger | thanks, anteaya | 20:34 |
| *** IlyaG has quit IRC | 20:34 | |
| anteaya | welcome | 20:34 |
| * AJaeger waves good-bye | 20:35 | |
| anteaya | bye | 20:35 |
| anteaya | thanks AJaeger | 20:35 |
| anteaya | see you tomorrow | 20:35 |
| mtreinish | jeblair: puppet just updated subunit2sql: http://puppetdb.openstack.org/report/subunit-worker01.openstack.org/faa5a2130bde54713e44a2c97704affe256b344d | 20:35 |
| AJaeger | talk to you tomorrow, anteaya ! | 20:35 |
| mtreinish | so we can restart the worker and hopefully it'll all work correctly | 20:35 |
| *** annegentle has joined #openstack-infra | 20:35 | |
| mordred | fungi: wow. I just ran autopep8 on shade and it only made 3 changes | 20:35 |
| mordred | fungi: http://paste.openstack.org/show/489046 | 20:35 |
| anteaya | nice | 20:36 |
| *** rcernin has quit IRC | 20:36 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:37 |
| mtreinish | infra-root: ^^^ any of you can help with that part :) | 20:37 |
| *** ybathia has joined #openstack-infra | 20:38 | |
| pleia2 | mtreinish: yeah, I can take care of this again | 20:38 |
| mtreinish | pleia2: cool, thanks | 20:38 |
| pleia2 | mtreinish: same as yesterday? | 20:38 |
| mtreinish | pleia2: yep, although this time hopefully we won't have to stop it after a couple min | 20:39 |
| *** claudiub has quit IRC | 20:39 | |
| fungi | (and then run around with a fire extinguisher) | 20:39 |
| *** annegentle has quit IRC | 20:40 | |
| pleia2 | mtreinish: running | 20:40 |
| mtreinish | pleia2: well we've got multiple runs already, so that's a good sign :) | 20:40 |
| cj | mordred: hmm... one of the nice things about perltidy is ~/.perltidyrc, which can alter the minutia about how formatting is implemented | 20:40 |
| pleia2 | mtreinish: as you saw yesterday, I don't think the filter is working though, since it's still looking at everything | 20:40 |
| *** maishsk has quit IRC | 20:41 | |
| pleia2 | mtreinish: so it tries to grab testrepository.subunit and gets a 404 a lot | 20:41 |
| *** sripriya__ has quit IRC | 20:41 | |
| pleia2 | but I'm not seeing the IO failures, so yay :) | 20:42 |
| *** rockyg has quit IRC | 20:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Add groups for Training labs https://review.openstack.org/283493 | 20:42 |
| *** maishsk has joined #openstack-infra | 20:42 | |
| mtreinish | pleia2: well it probabvly will still get 404s on runs that don't have subunit | 20:42 |
| mtreinish | either because they don't generate it, or because of an early failure | 20:42 |
| mtreinish | and also the gz vs non-gz thing will generate 404s too I think | 20:42 |
| mtreinish | we probably shouldn't log a stack trace and just do a single log message | 20:43 |
| *** ybathia has quit IRC | 20:43 | |
| sdague | AJaeger: thanks | 20:43 |
| pleia2 | mtreinish: ah, ok | 20:43 |
| mordred | cj: ok. I was wrong - pyfmt is horrible. autopep8, as fungi mentions, is pretty good though | 20:43 |
| *** annegentle has joined #openstack-infra | 20:43 | |
| fungi | i like that autopep8 mostly doesn't overreach | 20:43 |
| fungi | it reformats things that are safe to reformat, and leaves the hard decisions to you | 20:44 |
| openstackgerrit | Merged openstack-infra/project-config: Add proper groups for Training guides https://review.openstack.org/283453 | 20:44 |
| mtreinish | pleia2: you can see the little blue on the bottom right of the first graph: http://status.openstack.org/openstack-health/#/ | 20:44 |
| mtreinish | that matches up to how many runs we've worked through... | 20:44 |
| fungi | that said, i wouldn't use it in automation personally. i've really only found it useful for cleaning up some of my older and larger codebases which needed modernizing | 20:44 |
| mordred | yup | 20:45 |
| *** david-lyle has quit IRC | 20:45 | |
| fungi | and from that point on, i style-check all my commits | 20:45 |
| *** maishsk has quit IRC | 20:45 | |
| pleia2 | mtreinish: oh good :) | 20:46 |
| mtreinish | pleia2: is it still going ok? I don't see any new ones being added | 20:46 |
| pleia2 | mtreinish: yep | 20:46 |
| sdague | cj: well, with python there isn't really different standards, because whitespace counts | 20:47 |
| mtreinish | hmm, ok | 20:47 |
| sdague | my recommendation, get flake8 running in your emacs buffer under flymake, and it will just tell you when things are wrong in buffer | 20:47 |
| *** david-lyle has joined #openstack-infra | 20:47 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Move networking-odl rally job into check. https://review.openstack.org/282591 | 20:48 |
| mtreinish | pleia2: ah, its just sluggish (or I'm impatient) | 20:48 |
| *** roxanaghe has joined #openstack-infra | 20:49 | |
| *** |-paul-| has joined #openstack-infra | 20:50 | |
| lifeless | mtreinish: oh great | 20:50 |
| mordred | sdague: line wrapping and continuation style are still left - but I agree with you - it's mostly just the-one-style | 20:51 |
| *** gyee has quit IRC | 20:52 | |
| *** rockyg has joined #openstack-infra | 20:52 | |
| *** rossella_s has quit IRC | 20:53 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in deploy.sh https://review.openstack.org/287433 | 20:54 |
| *** rossella_s has joined #openstack-infra | 20:54 | |
| *** kencjohnston has quit IRC | 20:55 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: set -o pipefail in deploy.sh https://review.openstack.org/287433 | 20:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:56 |
| cj | thanks, mordred. I'm feeding the source to autopep8 now and cleaning up the exceptional cases. Can someone tell me how to make a print statement python3 compatable? Wrap () around the arguments? | 20:57 |
| *** pvaneck has quit IRC | 20:57 | |
| *** maishsk has joined #openstack-infra | 20:57 | |
| *** pvaneck has joined #openstack-infra | 20:58 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/285901 | 20:59 |
| *** apoorvad has joined #openstack-infra | 20:59 | |
| *** kencjohnston has joined #openstack-infra | 20:59 | |
| *** kgiusti has left #openstack-infra | 21:00 | |
| *** korzen has joined #openstack-infra | 21:01 | |
| *** sridhar_ram1 has joined #openstack-infra | 21:01 | |
| keedya | AJaeger: can I get a review on https://review.openstack.org/#/c/284827/ | 21:02 |
| *** _ody has quit IRC | 21:03 | |
| *** sridhar_ram has quit IRC | 21:03 | |
| *** _ody has joined #openstack-infra | 21:03 | |
| mordred | cj: yes | 21:04 |
| anteaya | cj: looks like yes: https://docs.python.org/3.0/whatsnew/3.0.html#print-is-a-function | 21:04 |
| *** abregman|afk has quit IRC | 21:07 | |
| *** arxcruz has joined #openstack-infra | 21:08 | |
| *** aysyd has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Disable image uplaods to vexxhost" https://review.openstack.org/287389 | 21:09 |
| *** doug-fish has quit IRC | 21:09 | |
| *** moravec has joined #openstack-infra | 21:09 | |
| *** abregman has joined #openstack-infra | 21:10 | |
| openstackgerrit | Artur Korzeniewski proposed openstack-infra/project-config: Create DVR multinode grenade job for Neutron upgrade tests https://review.openstack.org/250215 | 21:10 |
| keedya | Hi All, I know you are busy!! I have a new project that i am trying to push to openstack. Is there a chance to get reviews ? | 21:10 |
| *** julim has quit IRC | 21:11 | |
| korzen | AJaeger, I have addressed your comment in https://review.openstack.org/#/c/250215 | 21:11 |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Increase timeout for yum install of development tools https://review.openstack.org/274087 | 21:13 |
| *** ldnunes has quit IRC | 21:14 | |
| *** doug-fish has joined #openstack-infra | 21:15 | |
| *** doug-fis_ has joined #openstack-infra | 21:16 | |
| *** korzen has quit IRC | 21:17 | |
| jhesketh | Morning | 21:18 |
| *** aeng has quit IRC | 21:18 | |
| anteaya | fungi AJaeger http://paste.openstack.org/show/489054/ so I figured out why the output was saying 8500 jobs and my grep had 12000 jobs, the jobs are in both check and gate, many of them anyway | 21:19 |
| anteaya | morning jhesketh | 21:19 |
| anteaya | fungi AJaeger so the percentage of non-voting jobs in check is 15% | 21:19 |
| *** doug-fish has quit IRC | 21:20 | |
| anteaya | we have 15 non-voting jobs in the gate queue and I will look at cleaning those up | 21:20 |
| *** aeng has joined #openstack-infra | 21:20 | |
| openstackgerrit | Travis Tripp proposed openstack/requirements: Add python-searchlightclient>=0.2.0 https://review.openstack.org/268394 | 21:23 |
| *** doug-fish has joined #openstack-infra | 21:23 | |
| *** amrith is now known as _amrith_ | 21:24 | |
| *** doug-fis_ has quit IRC | 21:25 | |
| *** smarcet has quit IRC | 21:25 | |
| *** dkranz has quit IRC | 21:27 | |
| *** dkranz has joined #openstack-infra | 21:30 | |
| *** alivigni has quit IRC | 21:30 | |
| openstackgerrit | gordon chung proposed openstack/requirements: bump ceilometermiddleware upper constraints https://review.openstack.org/287416 | 21:34 |
| *** doug-fis_ has joined #openstack-infra | 21:35 | |
| *** reed_ has quit IRC | 21:35 | |
| *** baoli has quit IRC | 21:38 | |
| *** ZZelle_ has joined #openstack-infra | 21:38 | |
| *** doug-fish has quit IRC | 21:38 | |
| openstackgerrit | abregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 21:39 |
| clarkb | something something success https://jenkins01.openstack.org/job/gate-puppet-glance-puppet-unit-3.7-dsvm-centos7/68/consoleFull | 21:41 |
| clarkb | mnaser: ^ you are official providing test resources now | 21:41 |
| clarkb | mnaser: are you ok with us bumping the number of servers we use up to 25 now? | 21:41 |
| mnaser | oh sweet, lets go for it if things are going well clarkb | 21:42 |
| pabelanger | clarkb: mnaser http://grafana.openstack.org/dashboard/db/nodepool-vexxhost is also a thing too | 21:42 |
| mnaser | pabelanger: ive been eyeing it for sometime :) interesting metrics | 21:43 |
| mnaser | the most interesting one for me being error node launch attempts | 21:43 |
| mnaser | we track this already internally anyways | 21:43 |
| *** doug-fish has joined #openstack-infra | 21:43 | |
| *** dizquierdo has quit IRC | 21:43 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Make Packstack integration jobs voting, add to gate https://review.openstack.org/287461 | 21:44 |
| pabelanger | mnaser: ya, so far nothing reported, which is good | 21:44 |
| *** dizquierdo has joined #openstack-infra | 21:44 | |
| mnaser | i'm hoping at 1 instance nothing breaks :-P | 21:44 |
| *** angdraug has quit IRC | 21:45 | |
| dmsimard | ^ AJaeger, anteaya, pabelanger, ianw et al, thanks for your help. Packstack integration jobs are ready to gate ! | 21:45 |
| clarkb | the one thing I wnat to double check is that we have a job that ran and used our mirror with no problems | 21:46 |
| *** doug-fis_ has quit IRC | 21:46 | |
| pabelanger | dmsimard: good work! I need to spend some time looking at what you did | 21:46 |
| *** erlon has quit IRC | 21:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use all our vexxhost quota https://review.openstack.org/287462 | 21:47 |
| *** jsavak has quit IRC | 21:47 | |
| clarkb | so will look into that while ^ gets through check testing | 21:47 |
| dmsimard | pabelanger: nothing special (in project-config) | 21:47 |
| dmsimard | pabelanger: but happy to discuss nonetheless :) | 21:47 |
| *** jsavak has joined #openstack-infra | 21:47 | |
| dmsimard | I'm writing an e-mail to rdo-list and openstack-dev right now about it | 21:47 |
| *** |-paul-| has quit IRC | 21:47 | |
| mnaser | clarkb: i setup the right quota but if you'd be kind enough to double check it (if you have not). that would be useful | 21:47 |
| clarkb | https://jenkins01.openstack.org/job/gate-tempest-dsvm-full/36400/console used the mirror just fine so thats good | 21:48 |
| *** krtaylor has quit IRC | 21:48 | |
| clarkb | mnaser: I did look, if -1 means unlimited then I think our ram quota is ultimately what determines our instance quota and if I mathed right you got it at 25 8GB instances | 21:48 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Remove non-voting jobs from the gate pipeline. https://review.openstack.org/287464 | 21:48 |
| *** sc68cal has quit IRC | 21:48 | |
| mnaser | exactly how i set things up so we're good | 21:48 |
| *** matrohon has quit IRC | 21:49 | |
| *** piet has quit IRC | 21:49 | |
| jeblair | mnaser: has someone pointed http://grafana.openstack.org/dashboard/db/nodepool-vexxhost to you yet? | 21:49 |
| *** jtomasek has quit IRC | 21:50 | |
| fungi | oh, cool, that's live now | 21:50 |
| *** esikache1 has joined #openstack-infra | 21:50 | |
| mnaser | jeblair: yes, waiting to see the number go up soon | 21:50 |
| jeblair | oh heh, i see pabelanger beat me to it just a few mins ago :) | 21:50 |
| *** sc68cal has joined #openstack-infra | 21:50 | |
| anteaya | EmilienM mtreinish can you take a look at https://review.openstack.org/#/c/287464/1 please? it removes non-voting jobs from the gate pipeline | 21:51 |
| EmilienM | anteaya: sure thing | 21:51 |
| anteaya | EmilienM: thank you | 21:51 |
| mtreinish | anteaya: can you add me on the review, so I'll have an email to remind me | 21:52 |
| anteaya | sure | 21:52 |
| mtreinish | I'm about to head home and I'll lose the link | 21:52 |
| anteaya | mtreinish: yup, will do | 21:52 |
| anteaya | thank you | 21:52 |
| EmilienM | nibalizer: please also look | 21:53 |
| fungi | we're still somewhat frequently hitting ssh timeouts when trying to bring up nodes in bluebox-sjc1 | 21:55 |
| *** armax has quit IRC | 21:55 | |
| *** esikache1 has quit IRC | 21:55 | |
| fungi | not sure what's going on there | 21:55 |
| EmilienM | anteaya: done | 21:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 21:55 |
| anteaya | EmilienM: thank you | 21:55 |
| fungi | also nodepoold is unhappy about being unable to clean up old images from infracloud-west... should those just be deleted from the database? | 21:56 |
| fungi | or ignore? | 21:56 |
| anteaya | EmilienM: all the jobs I removed are nonvoting | 21:56 |
| anteaya | according to the zuul layout output | 21:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add total singlestats for nodepool dashboard https://review.openstack.org/287469 | 21:56 |
| jeblair | fungi: i think just rm them | 21:56 |
| pabelanger | jeblair: ^ add total counts for build, ready, etc to top level nodepool page | 21:57 |
| EmilienM | anteaya: look https://review.openstack.org/#/c/286845/ | 21:57 |
| anteaya | EmilienM: if they should be voting then perhaps you need to removing some voting; false statements | 21:57 |
| EmilienM | only gate-puppet-openstack-integration-scenario003-tempest-dsvm-trusty is not voting | 21:57 |
| fungi | jeblair: oh, yeah, no chance of them ever coming back as referenced anyway, what with a planned rename | 21:57 |
| anteaya | not according to the zuul layout output | 21:57 |
| mordred | clarkb: I'm seeing a lot of node fails from ovh and bluebox | 21:58 |
| anteaya | EmilienM: which is what I was going by | 21:58 |
| mordred | clarkb: I checked bluebox floating ips and we don't seem to be leaked | 21:58 |
| clarkb | mordred: nova show the hosts is usually where I start if the fails aren't obvious | 21:58 |
| clarkb | mordred: and sometimes nova console log | 21:58 |
| EmilienM | anteaya: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1090-L1098 | 21:58 |
| mordred | clarkb: yah - nova shows a set of hosts sitting there happily | 21:59 |
| *** armax has joined #openstack-infra | 21:59 | |
| *** IlyaG has joined #openstack-infra | 21:59 | |
| clarkb | mordred: is ssh failing to them? | 21:59 |
| anteaya | if you run tox -e zuul and capture the output and then use the file from INFO:zuul.DependentPipelineManager:Configured Pipeline Manager gate to the post pipeline | 21:59 |
| clarkb | console-log can be helpful for diagnosing that if glean/cloud-init failed | 21:59 |
| anteaya | that output shows the 001 scenario for trusty as non-voting | 22:00 |
| anteaya | so I'm not sure what is happening here | 22:00 |
| fungi | i guess the snapshot_image table is not very accurately named these days, since it also covers the uploaded copies of dib images | 22:00 |
| EmilienM | anteaya: look https://review.openstack.org/#/c/286845/ | 22:00 |
| anteaya | yeah I know, I did | 22:00 |
| fungi | jeblair: delete from snapshot_image where provider_name="infracloud-west"; | 22:00 |
| EmilienM | they are voting | 22:00 |
| fungi | lgty? | 22:01 |
| mordred | clarkb: they mostly look like building or used state | 22:01 |
| anteaya | I'm saying there is a mismatch | 22:01 |
| anteaya | between what zuul is reporting on output as nonvoting | 22:01 |
| clarkb | mordred: they are erorring after being used? | 22:01 |
| anteaya | and what is running as nonvoting | 22:01 |
| fungi | jeblair: the equivalent select returns 4 rows which match the image ids it's complaining about. anything else i need to clean up after that? | 22:01 |
| mordred | clarkb: ah - well | 22:01 |
| mordred | clarkb: cloud-init-nonet[13.37]: waiting 120 seconds for network device | 22:01 |
| mordred | clarkb: I do see that on one in the building state | 22:01 |
| *** thorst has quit IRC | 22:01 | |
| mnaser | signs of a dead dhcp server ^ | 22:02 |
| mordred | jlk: heya! so ^^ | 22:02 |
| *** thorst has joined #openstack-infra | 22:02 | |
| mordred | jlk: we're seeing a bunch of bluebox fails | 22:02 |
| jlk | crap. | 22:02 |
| mnaser | or ovs flows gone poof from compute nodes | 22:02 |
| jlk | no ovs | 22:02 |
| mordred | mnaser: they're using linuxbridge, so it sholdn't be the second :) | 22:02 |
| *** dprince has quit IRC | 22:03 | |
| jlk | let me ping my ops golks | 22:03 |
| jlk | folks | 22:03 |
| mnaser | gotcha | 22:03 |
| mordred | cool | 22:03 |
| mordred | jlk: also, if you do find something and it's somehting I could have found/done from my admin panel without bugging y'all, that's also potentially useful info | 22:03 |
| anteaya | EmilienM: here is the tox -e zuul output for puppet-cinder's gate: http://paste.openstack.org/show/489062/ | 22:03 |
| jlk | mordred: probably going ot have to restart services, which I don't think you can do | 22:04 |
| mordred | jlk: kk | 22:04 |
| jlk | mordred: are ou seeing instance build failures, or some other failure? | 22:04 |
| mordred | jlk: eba2a2fb-906c-449d-8143-08646fde6ed3 is a uuid of a host in that state that I've held | 22:04 |
| mordred | jlk: yah I get cloud-init-nonet[13.37]: waiting 120 seconds for network device in the console log | 22:04 |
| mordred | jlk: which doesn't seem happy | 22:04 |
| *** sridhar_ram1 has quit IRC | 22:05 | |
| *** BobBall_AWOL has joined #openstack-infra | 22:05 | |
| mordred | jlk: oh - that one finally came up | 22:05 |
| jlk | actually up, and pingable, or just gave up on cloud-init? | 22:06 |
| *** achanda has quit IRC | 22:06 | |
| mordred | jlk: yup. that one is now actually up. sorry - I thought I'd caught one in unhpapy state | 22:06 |
| *** jsavak has quit IRC | 22:06 | |
| mordred | jlk: lemme keep trying to catch one | 22:06 |
| *** thorst has quit IRC | 22:06 | |
| *** sridhar_ram has joined #openstack-infra | 22:06 | |
| jlk | okay | 22:07 |
| mordred | jlk: hahahaha. literally 10 minutes ago the bluebox failure rate dropped off and became normal again | 22:07 |
| mordred | jlk: so, like, the instant I started investigating and bothering you | 22:07 |
| timrc | mordred: quantum debugging | 22:08 |
| fungi | i'm still trying to catch one of the boots in ovh-gra1 that ends up in error state according to nova, but they're gone from the instance list by the time i can plug in the uuid and run nova show | 22:08 |
| fungi | like trying to catch fish with your bare hands | 22:08 |
| *** BobBall_1WOL has quit IRC | 22:08 | |
| *** _joes_ has quit IRC | 22:09 | |
| jlk | lolol | 22:09 |
| mordred | fungi: maybe we should add some stuff to nodepool to grab a few things like that when it deletes due to error and log them | 22:09 |
| fungi | yeah, this is one of the few times i miss rackspace's "we'll delete the instance when we get around to it" asynchronicity | 22:10 |
| jlk | only alert we got was that one of the nodes saw a heavy amount of network traffic. | 22:10 |
| jlk | a compute node though. | 22:10 |
| *** doug-fis_ has joined #openstack-infra | 22:10 | |
| fungi | hrm, we actually have a lot ending up in error in ovh-bhs1 too | 22:11 |
| *** doug-fish has quit IRC | 22:11 | |
| mnaser | storing the failed uuids is really useful from the operator side | 22:11 |
| mnaser | its really easy to figure out what happened on our side | 22:12 |
| mordred | fungi: so, looking at nodepool logs | 22:12 |
| fungi | yeah, we do log the uuids. it's just easier if i can spot more detail in the nova api as well | 22:12 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Enable pep8 job on tripleo-heat-templates https://review.openstack.org/258122 | 22:13 |
| mordred | fungi: I only see 78 ERROR to 2260 ACTIVE for bluebox | 22:13 |
| mordred | fungi: I see 1626 ERROR to 2834 active for ovh | 22:13 |
| mnaser | also, is there a policy in terms of using the data of the openstack-infra graphite instance (i'd like to bring the dashboard you use into our own grafana) | 22:13 |
| mordred | mnaser: nope - it's all public | 22:13 |
| fungi | mordred: yep | 22:13 |
| *** achanda has joined #openstack-infra | 22:13 | |
| fungi | it's pretty constant | 22:14 |
| mnaser | sweet. we'll add it to our dashboards | 22:14 |
| clarkb | just keep in mind we periodically change keys out or reorg the schema | 22:14 |
| *** e0ne has quit IRC | 22:14 | |
| *** maishsk has quit IRC | 22:14 | |
| mordred | mnaser: we mostly just would prefer you don't use any of our graphs as comparative marketing materials "we're 100X faster than XXX as shown by OpenStack Infra" | 22:14 |
| clarkb | but that is fairly infrequent | 22:14 |
| mnaser | mordred: nah, that's not what we're after, we just want to have the visibility in our own grafana instance | 22:14 |
| mordred | ++ | 22:15 |
| mnaser | because our eyes are on it more than the one you have :P | 22:15 |
| fungi | mordred: nova list in ovh-gra1 consistently turns up at least a few instances in "ERROR ... NOSTATE" though | 22:15 |
| pabelanger | mnaser: consider using grafyaml to build your dashboards :) | 22:15 |
| mordred | mnaser: have you see the grafyaml project pabelanger put together? | 22:15 |
| mordred | jinx | 22:15 |
| *** sdake_ has joined #openstack-infra | 22:15 | |
| fungi | different instances each time since they're getting deleted almost instantly | 22:15 |
| mnaser | mordred lol | 22:15 |
| pabelanger | https://github.com/openstack-infra/project-config/tree/master/grafana | 22:15 |
| pabelanger | mordred: moar people on grafyaml | 22:15 |
| * mnaser looks | 22:16 | |
| anteaya | hopefully using our git mirrors | 22:16 |
| fungi | github.com is one of our git mirrors, just one we don't have much control over :/ | 22:16 |
| EmilienM | anteaya: I don't understand :) | 22:17 |
| fungi | so not one of our git servers | 22:17 |
| *** e0ne has joined #openstack-infra | 22:17 | |
| dmsimard | mordred: did you figure out that cloud-init nonet issue ? | 22:18 |
| ianw | dmsimard: cool ... so i think you want to modify the template in that change | 22:18 |
| *** sdake has quit IRC | 22:18 | |
| anteaya | EmilienM: yeah, I don't either, I'm confused about why zuul output is showing nonvoting on those jobs yet treating them as voting | 22:18 |
| mordred | dmsimard: it was just time - it has decided to resolve itself | 22:18 |
| anteaya | fungi: sorry you are correct, our git servers | 22:18 |
| dmsimard | ianw: thanks, feel free to comment, I'm a noob at this | 22:18 |
| mordred | dmsimard: so I'm going to add more debugging | 22:18 |
| dmsimard | mordred: ok, just in case, shameless plug https://dmsimard.com/2014/05/02/no-network-on-ubuntu-14-04-cloud-image-with-cloud-init/ | 22:18 |
| anteaya | EmilienM: I'm hoping AJaeger may have some insight after he takes a look | 22:19 |
| EmilienM | anteaya: cool no worries :-) | 22:19 |
| *** weshay has quit IRC | 22:19 | |
| dmsimard | mordred: I filed https://bugs.launchpad.net/cloud-init/+bug/1315501 slightly before Ubuntu Trusty came out I think.. and 16.04 is right around the corner | 22:19 |
| openstack | Launchpad bug 1315501 in cloud-init (Ubuntu) "cloud-init does not use interfaces.d in trusty" [High,Confirmed] | 22:19 |
| mordred | dmsimard: nice | 22:19 |
| mordred | dmsimard: so - that'll also go away from us once we're on our new ubuntu-trusty images everywhere because we don't use cloud-init on those at all | 22:20 |
| anteaya | EmilienM: if the nonvoting attribute is not accurately applied via tox then our foray into finding percentage of nonvoting jobs is completely inaccurate | 22:20 |
| mordred | :) | 22:20 |
| *** achanda has quit IRC | 22:20 | |
| dmsimard | mordred: cool :D | 22:20 |
| *** ybathia has joined #openstack-infra | 22:20 | |
| anteaya | EmilienM: as that was what I was attempting to do when I uncovered what I believed to be nonvoting jobs in the gate pipeline | 22:20 |
| mordred | dmsimard: shameless plug from my end: http://git.openstack.org/cgit/openstack-infra/glean :) | 22:20 |
| dmsimard | mordred: that looks sexy | 22:21 |
| clarkb | fungi: looking at nodepool logs we are building ubuntu-trusty with the compat option set | 22:21 |
| fungi | clarkb: odd, maybe that's a default then and the addition in nodepool.yaml is a no-op? | 22:22 |
| *** claudiub has joined #openstack-infra | 22:22 | |
| clarkb | fungi: ya dib may be doing it for us | 22:22 |
| fungi | i wonder if it didn't always and we added those as a workaround while waiting for the behavior change upstream | 22:23 |
| fungi | so there goes that theory | 22:23 |
| clarkb | which means the differences are ubuntu vs ubuntu-minimal+simple-init+growroot | 22:23 |
| anteaya | mtreinish: thank you | 22:23 |
| clarkb | if I had to guess based on that growroot is a likely culprit | 22:23 |
| *** thorst has joined #openstack-infra | 22:23 | |
| mtreinish | anteaya: sure, np | 22:23 |
| fungi | clarkb: which doesn't seem like it should impact the contents of /opt | 22:23 |
| fungi | clarkb: or did /opt size turn out to be consistent between both images? | 22:23 |
| clarkb | fungi: I think /opt is the same in both cases (though I havne't double checked) | 22:23 |
| mtreinish | anteaya: tbh I was kinda surprised the forward jobs were nv on gate | 22:23 |
| anteaya | mtreinish: me too | 22:24 |
| anteaya | did you see the rule in the top of zuul/layout.yaml? | 22:24 |
| *** claudiub has quit IRC | 22:24 | |
| clarkb | I am going to try some local experimentation | 22:24 |
| *** claudiub has joined #openstack-infra | 22:24 | |
| anteaya | mtreinish: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1232 | 22:25 |
| openstackgerrit | C.J. Collier proposed openstack-infra/gerritbot: Improve gerritbot error handling https://review.openstack.org/287056 | 22:25 |
| *** cbader has joined #openstack-infra | 22:25 | |
| mtreinish | anteaya: yeah, that job only runs on stable branches, instead of doing n-1 -> n it does n -> n+1 | 22:26 |
| mtreinish | but it's never really worked which is why it's nv | 22:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Add an error log with the server fault message https://review.openstack.org/287481 | 22:27 |
| *** e0ne has quit IRC | 22:27 | |
| *** thorst has quit IRC | 22:27 | |
| mordred | fungi: ^^ maybe that gets us slightly more log information | 22:27 |
| clarkb | oh cache-bindep | 22:28 |
| clarkb | maybe it really is the packages | 22:28 |
| *** amitgandhinz has quit IRC | 22:28 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Make Packstack integration jobs voting, add to gate https://review.openstack.org/287461 | 22:28 |
| anteaya | mtreinish: okay thanks for the +1 | 22:28 |
| mordred | clarkb: cache bindep | 22:28 |
| mordred | yah | 22:28 |
| clarkb | I take that back both cache bindep | 22:28 |
| mordred | oh | 22:28 |
| *** dkranz has quit IRC | 22:28 | |
| *** cbader has quit IRC | 22:28 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 22:29 |
| dmsimard | ianw: I think this might be better now: https://review.openstack.org/#/c/287461/2/ thanks for your feedback again :) | 22:29 |
| clarkb | I am trying a local build without growroot | 22:29 |
| fungi | clarkb: right. that was my initial assumption but 1. the apt cache shouldn't be anywhere near that huge and 2. turns out we were running it for both images | 22:30 |
| *** jamesmcarthur has quit IRC | 22:30 | |
| *** jamesmcarthur has joined #openstack-infra | 22:31 | |
| *** edmondsw has quit IRC | 22:32 | |
| *** jamesmcarthur has quit IRC | 22:33 | |
| ianw | dmsimard: so yeah, that's not going to work. left a comment, but you just want to duplicate what was there into a "gate:" section, and remove the voting regex | 22:34 |
| fungi | mordred: if getServerFromList() contains a 'fault' key then it always has a value which is a dict with a 'message' key? | 22:34 |
| dmsimard | ianw: oh, why not ? the variables are there for it ? https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L5123-L5134 | 22:35 |
| *** jamesmcarthur has joined #openstack-infra | 22:35 | |
| dmsimard | oh, saw your comment, ok | 22:35 |
| fungi | mordred: or do we need to guard against the possibility of resource['fault']['message'] raising an exception? | 22:35 |
| mordred | fungi: I have code in shade which just prints the sever['fault']['message'] on error | 22:35 |
| fungi | okay | 22:35 |
| mordred | fungi: I put in the if because sometimes these are images | 22:35 |
| *** _amrith_ is now known as amrith | 22:36 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Make Packstack integration jobs voting, add to gate https://review.openstack.org/287461 | 22:36 |
| ianw | dmsimard: yeah, that {name} is just zuul substituting in the name: of that particular job setup. jjb are separate templates | 22:36 |
| fungi | mordred: so basically that should never raise? lgtm then | 22:36 |
| *** esker has joined #openstack-infra | 22:36 | |
| ianw | dmsimard: cool, LGTM, just let jenkins churn away at it | 22:37 |
| *** ayoung has quit IRC | 22:37 | |
| dmsimard | yup ok | 22:38 |
| *** Kaiyan has quit IRC | 22:38 | |
| *** jpr has quit IRC | 22:38 | |
| mnaser | how often does nodepool upload images? | 22:38 |
| *** Sukhdev has joined #openstack-infra | 22:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 22:39 |
| *** doug-fis_ has quit IRC | 22:39 | |
| clarkb | mnaser: once a day | 22:40 |
| ianw | clarkb: sometimes :) speaking of ... has the rax uploads been restarted? | 22:40 |
| mnaser | clarkb: i see, depending on how this goes, it may be ideal for us to shuffle around glance-api to make it upload faster, and cache at each az so access/time to ready gets faster | 22:41 |
| ianw | i did check a recent f23 run and it didn't seem to have build timestamps | 22:41 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Make Packstack integration jobs voting, add to gate https://review.openstack.org/287461 | 22:41 |
| clarkb | ianw: they ran and failed today | 22:41 |
| *** annegentle has quit IRC | 22:41 | |
| *** doug-fish has joined #openstack-infra | 22:42 | |
| ianw | clarkb: failed to upload? | 22:42 |
| clarkb | ya I think with the same issue but I can dig up tracebacks if it helps | 22:42 |
| clarkb | (I should state that the uploads in the extra builders I started to get to vexxhost faster did succeed becuase they all requested new tokens | 22:43 |
| ianw | clarkb: well, i think best to retry with the two changes to shade we've got there when mordred releases ... pretty convinced that's it | 22:43 |
| clarkb | ianw: kk | 22:44 |
| *** doug-fis_ has joined #openstack-infra | 22:44 | |
| clarkb | thats easy enough to do once shade releases | 22:44 |
| mordred | clarkb, ianw: holding off on a shade release because of FF | 22:44 |
| mordred | but it's ready to tag/release whenever we're ready to do that | 22:44 |
| ianw | mordred: do i want to know what FF is? | 22:44 |
| clarkb | is http://tarballs.openstack.org/trove/images/ubuntu/mysql.qcow2 still a thing we use in tests? | 22:45 |
| * clarkb is just looking at general disk consumption and ^ jumps out | 22:45 | |
| *** xyang1 has quit IRC | 22:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Use all our vexxhost quota https://review.openstack.org/287462 | 22:46 |
| fungi | clarkb: i take it that's in the devstack file download links list? | 22:46 |
| clarkb | fungi: yup | 22:46 |
| ianw | that's a good question ... | 22:46 |
| fungi | ianw: feature freeze (week) | 22:47 |
| *** doug-fish has quit IRC | 22:47 | |
| anteaya | ianw: http://releases.openstack.org/mitaka/schedule.html | 22:48 |
| *** mriedem has quit IRC | 22:48 | |
| *** doug-fis_ has quit IRC | 22:48 | |
| mnaser | clarkb: so now that the change above has been merged, will the nodepool configs update eventually on next puppet run or something? | 22:49 |
| ianw | fungi: oh, doh, of course ... there's all these providers who i've never heard of, i thought it might be short-form for some new one :) | 22:49 |
| clarkb | mnaser: yup they will update on the next puppet run | 22:49 |
| clarkb | mnaser: and nodepool will figure out its config updated and start using it at that point | 22:49 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Remove non-voting jobs from the gate pipeline. https://review.openstack.org/287464 | 22:51 |
| openstackgerrit | Deklan Dieterly proposed openstack-infra/project-config: Add freezer-api job for tempest tests https://review.openstack.org/287486 | 22:52 |
| *** annegentle has joined #openstack-infra | 22:52 | |
| *** krtaylor has joined #openstack-infra | 22:52 | |
| anteaya | EmilienM: the patch you showed me with those jobs as voting are check jobs | 22:53 |
| anteaya | EmilienM: have you any gate jobs results? | 22:53 |
| anteaya | EmilienM: it is the gate queue I'm looking at | 22:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable bindep / devstack cache for fedora22 nodes https://review.openstack.org/287488 | 22:53 |
| EmilienM | anteaya: https://review.openstack.org/#/c/286788/ | 22:53 |
| EmilienM | look at gate jobs | 22:53 |
| EmilienM | anteaya: maybe you found a bug :-) | 22:54 |
| clarkb | mnaser: looks like our usage just jumped | 22:54 |
| clarkb | mnaser: quite a few instances building now | 22:54 |
| mnaser | building 23 | 22:54 |
| * mnaser watches | 22:54 | |
| anteaya | EmilienM: it is possible | 22:55 |
| *** sc68cal_ has joined #openstack-infra | 22:56 | |
| *** dims_ has quit IRC | 22:56 | |
| mnaser | of course the az with the 10gig networking instances go up super fast, the others are still getting setup | 22:58 |
| mnaser | i think we'll do a glance-api cache on the second zone soon to speed it up as well as local nova image cache will help for new deploys | 22:58 |
| *** sridhar_ram has quit IRC | 22:58 | |
| *** sc68cal has quit IRC | 22:58 | |
| *** annegentle has quit IRC | 22:59 | |
| ianw | clarkb: so, after a bit of poking, it seems there's "trove-integration" jobs which actually build that vm on the fly. i'm pretty convinced it's unused. i'll proposal a removal and get some reviews | 22:59 |
| *** jamesmcarthur has quit IRC | 22:59 | |
| *** Jeffrey4l has joined #openstack-infra | 23:00 | |
| clarkb | ianw: thank you | 23:00 |
| *** alivigni has joined #openstack-infra | 23:00 | |
| *** jamesmcarthur has joined #openstack-infra | 23:00 | |
| ianw | clarkb: although, it probably will hang around in old branches ... have to make sure to backport it to really get rid of it. | 23:01 |
| mnaser | clarkb all 25 in-use, zero errors on launch | 23:01 |
| mnaser | success.gif | 23:01 |
| clarkb | ianw: ya and older branches may need it so it won't die immediatley but this is step one in cleaning it up | 23:02 |
| *** jcoufal has quit IRC | 23:02 | |
| pabelanger | mnaser: nice | 23:02 |
| mnaser | i think time to ready should go down, i would def rather have instances cooked up in a faster time period | 23:02 |
| ddieterly | can i get some help with https://review.openstack.org/287486? | 23:02 |
| anteaya | mnaser: nice work | 23:02 |
| ddieterly | trying to get tempest tests configured to run | 23:03 |
| openstackgerrit | C.J. Collier proposed openstack-infra/gerritbot: Improve gerritbot error handling https://review.openstack.org/287056 | 23:03 |
| ddieterly | not sure if the gate_hook function will interfere with the tempest tests | 23:03 |
| ddieterly | looks like the tempest tests are driven by env vars | 23:03 |
| clarkb | mnaser: part of that may be our data not being entirely truthful, we process the servers serially iirc | 23:03 |
| clarkb | mnaser: I would have to double check in the code on that | 23:04 |
| *** dingyichen has joined #openstack-infra | 23:04 | |
| *** _sarob has joined #openstack-infra | 23:05 | |
| *** Qiming has joined #openstack-infra | 23:05 | |
| *** _sarob has quit IRC | 23:05 | |
| fungi | wow, i go 3 days (3!) without looking at my github spam and we've added 45 new repos to gerrit in that short time | 23:05 |
| mnaser | clarkb: i see | 23:05 |
| *** _sarob has joined #openstack-infra | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:07 | |
| openstackgerrit | Erno Kuvaja proposed openstack/requirements: Bump of glance-store to 0.13.0 https://review.openstack.org/287496 | 23:07 |
| *** sarob has quit IRC | 23:08 | |
| *** Sukhdev has quit IRC | 23:08 | |
| *** hashar has quit IRC | 23:09 | |
| *** gordc has quit IRC | 23:09 | |
| *** jamesmcarthur has joined #openstack-infra | 23:10 | |
| openstackgerrit | Max V.K. proposed openstack-infra/jenkins-job-builder: upd google-cloud-storage plugin support https://review.openstack.org/287497 | 23:10 |
| *** cznewt has quit IRC | 23:11 | |
| *** tiswanso has quit IRC | 23:12 | |
| *** yamamoto_ has joined #openstack-infra | 23:12 | |
| *** tiswanso has joined #openstack-infra | 23:12 | |
| *** tiswanso has quit IRC | 23:12 | |
| *** sobersabre has joined #openstack-infra | 23:13 | |
| *** thiagop has quit IRC | 23:13 | |
| *** tongli has quit IRC | 23:14 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:15 | |
| *** regXboi has quit IRC | 23:17 | |
| *** ajmiller has quit IRC | 23:17 | |
| sobersabre | hi. I've updated google-cloud-storage plugin support for jenkins job builder, I've added the previous committer of that plugin as reviewer. what's the correct reviewers list? | 23:17 |
| *** ajmiller has joined #openstack-infra | 23:18 | |
| sobersabre | https://review.openstack.org/#/c/287497/1 | 23:18 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: [WIP] Add update_service() https://review.openstack.org/287504 | 23:18 |
| anteaya | sobersabre: https://review.openstack.org/#/admin/groups/194,members | 23:18 |
| *** salv-orlando has joined #openstack-infra | 23:19 | |
| *** achanda has joined #openstack-infra | 23:20 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: midonet: Rename rally job so that it matches regex for doc-only changes https://review.openstack.org/286956 | 23:21 |
| sobersabre | anteaya: thanks. | 23:21 |
| *** rockyg has quit IRC | 23:22 | |
| *** salv-orl_ has quit IRC | 23:22 | |
| sobersabre | fdegir: ping | 23:22 |
| pabelanger | fungi: have to run for a bit, but wanted to share some bindep failures on fedora-23 with your: http://paste.openstack.org/show/489071/ | 23:22 |
| pabelanger | fungi: seems like bindep is not properly selecting the packages for some reason | 23:23 |
| Nakato | AJaeger: With post constraints jobs, do you remember what the issue was? I've been taking a look at that lately and I've not been able to figure out where things were going wrong. | 23:23 |
| pabelanger | eg: uuid-dev is getting pulled in when only listed as platform:dpkg | 23:23 |
| pabelanger | have to run, but hope to hack on it later tonight | 23:23 |
| *** alivigni has quit IRC | 23:23 | |
| *** yamamoto_ has quit IRC | 23:24 | |
| fungi | pabelanger: the good news is that bindep is pretty easily debuggable, and has a lot of existing test framework in place so if you figure out how to fix it we should be able to pretty easily avoid regressing | 23:24 |
| *** dimtruck is now known as zz_dimtruck | 23:25 | |
| fungi | lots of mocked out interactions | 23:26 |
| fungi | but also we have functional testing jobs for it (via the fallback jobs) | 23:27 |
| *** cznewt has joined #openstack-infra | 23:27 | |
| *** achanda has quit IRC | 23:28 | |
| pabelanger | fungi: actually, I have a way for use to debug it in the gate | 23:28 |
| pabelanger | 1 sec | 23:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable bindep / devstack cache for fedora22 nodes https://review.openstack.org/287488 | 23:30 |
| *** ashleighfarnham has quit IRC | 23:30 | |
| pabelanger | fungi: ianw: enabling ^ will install bindep on fedora-22 but not cache the packages at DIB build time | 23:30 |
| pabelanger | we can then, once new fedora-23 dibs land, use the experimental bindep-fallback job to debug | 23:31 |
| *** pahuang has joined #openstack-infra | 23:31 | |
| pabelanger | I first tested with that patch avoid and worked as expected | 23:31 |
| *** Sukhdev has joined #openstack-infra | 23:31 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| clarkb | so I ran this exact job on vexxhost earlier and don't think its related to the cloud but http://logs.openstack.org/36/259236/44/check/gate-tempest-dsvm-full/cd9593c/logs/devstacklog.txt.gz#_2016-03-02_23_22_51_961 failed | 23:32 |
| clarkb | looks like horizon is trying to compress things? | 23:32 |
| mnaser | clarkb: any idea where we can find id's of error'd servers? i see one had an issue so id like to see what happened | 23:33 |
| clarkb | mnaser: we don't publish the nodepool logs but I can look them up for you | 23:33 |
| mnaser | if it's not too much of a hassle, im just curious | 23:34 |
| *** annegentle has joined #openstack-infra | 23:35 | |
| clarkb | mnaser: http://paste.openstack.org/show/489073/ thats one, looks like create server failed so we never got a uuid? you might be able to grep for that 8491827 number in your logs as it will be used in the hostname | 23:35 |
| mnaser | i suspect i know the cause (it shouldn't happen again). i'll keep an eye out | 23:36 |
| *** pahuang has quit IRC | 23:36 | |
| mnaser | it's probably a combination that it happened with a small route blip in ospf | 23:36 |
| clarkb | mnaser: http://paste.openstack.org/show/489074/ is similar no uuid becuase create server failed | 23:37 |
| mnaser | noticed a few inefficient traffic patterns so we're moving it around a bit | 23:37 |
| *** ddieterly has quit IRC | 23:39 | |
| *** caphrim007 has quit IRC | 23:47 | |
| *** pahuang has joined #openstack-infra | 23:49 | |
| *** sarob has joined #openstack-infra | 23:49 | |
| *** ayoung has joined #openstack-infra | 23:50 | |
| *** salv-orlando has quit IRC | 23:50 | |
| *** annegentle has quit IRC | 23:51 | |
| *** esikache1 has joined #openstack-infra | 23:51 | |
| *** salv-orlando has joined #openstack-infra | 23:51 | |
| *** pradk has quit IRC | 23:53 | |
| *** _sarob has quit IRC | 23:53 | |
| *** kencjohnston has quit IRC | 23:53 | |
| *** sarob has quit IRC | 23:54 | |
| *** Jeffrey4l has quit IRC | 23:55 | |
| tchaypo | http://www.theregister.co.uk/2016/03/01/new_jenkins_moves_into_alpha/ | 23:55 |
| *** esikache1 has quit IRC | 23:55 | |
| *** Jeffrey4l has joined #openstack-infra | 23:56 | |
| clarkb | something something hey jjb | 23:56 |
| tchaypo | It sounds like jenkins2.0 will have an equivalent of JJB - but it involves writing groovy instead of yaml. | 23:56 |
| mordred | yah | 23:56 |
| *** msuriar has quit IRC | 23:56 | |
| mordred | because that's what I want | 23:56 |
| pleia2 | hah | 23:56 |
| tchaypo | I'm going to go laugh at them | 23:56 |
| mordred | tchaypo: please do | 23:56 |
| *** msuriar has joined #openstack-infra | 23:57 | |
| mordred | tchaypo: consider me laughing in concert with you | 23:57 |
| clarkb | I just want the ssh slaves plugin to work | 23:57 |
| clarkb | that would alleviate about 100% of our current jenkins problems | 23:57 |
| *** arxcruz has quit IRC | 23:57 | |
| gus | Somewhere not at the top of my todo list is to go rewrite the input/templating part of jjb to use jsonnet instead. | 23:57 |
| gus | (particularly for the multiple template inheritance, and actual sane language semantics) | 23:57 |
| *** ajmiller has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!