| *** amotoki has joined #openstack-infra | 00:00 | |
| *** ashleighfarnham has quit IRC | 00:01 | |
| *** Jeffrey4l has joined #openstack-infra | 00:02 | |
| dougwig | proxy errors from gerrit, fyi. | 00:03 |
|---|---|---|
| *** Sukhdev has joined #openstack-infra | 00:03 | |
| dougwig | and it's back. :) | 00:03 |
| *** yamamoto_ has quit IRC | 00:06 | |
| *** maishsk has quit IRC | 00:07 | |
| *** maishsk has joined #openstack-infra | 00:08 | |
| *** r-daneel has quit IRC | 00:08 | |
| *** sabeen2 has quit IRC | 00:09 | |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: Added new project for the OSA role break out https://review.openstack.org/272331 | 00:11 |
| *** ybathia has joined #openstack-infra | 00:12 | |
| jeblair | dougwig: errors or delays at 00:00 UTC are database backups :( | 00:14 |
| *** jtomasek has quit IRC | 00:14 | |
| *** amotoki has quit IRC | 00:15 | |
| *** Qiming has quit IRC | 00:15 | |
| jeblair | mordred, krotscheck: the dkms patch automatically fixed gra1: http://mirror.gra1.ovh.openstack.org/pypi/ | 00:15 |
| jeblair | so we're up in rax and ovh | 00:15 |
| dougwig | jeblair: oh didn't notice it was 00:00. wow, las vegas and boise are in different timezones. thanks. | 00:16 |
| *** diana_clarke has joined #openstack-infra | 00:16 | |
| jeblair | dougwig: the rule is that no matter where in the world you are, when gerrit does db backups, it's time for beer | 00:16 |
| *** salv-orl_ has quit IRC | 00:17 | |
| dougwig | jeblair: i love that rule. | 00:17 |
| *** chlong has quit IRC | 00:20 | |
| *** chlong has joined #openstack-infra | 00:21 | |
| *** rfolco has joined #openstack-infra | 00:21 | |
| *** bodepd has joined #openstack-infra | 00:21 | |
| *** bodepd has quit IRC | 00:22 | |
| *** bodepd has joined #openstack-infra | 00:22 | |
| *** rbrndt has quit IRC | 00:24 | |
| *** dhp_ has quit IRC | 00:24 | |
| openstackgerrit | John L. Villalovos proposed openstack/requirements: Add openstack/bifrost to projects.txt https://review.openstack.org/272334 | 00:25 |
| *** rfolco has quit IRC | 00:26 | |
| jeblair | mordred: i ran shade-launch-node on hpcloud and got 'Server reached ACTIVE state without being allocated an IP address.' | 00:26 |
| *** hichihara has joined #openstack-infra | 00:27 | |
| *** jtomasek has joined #openstack-infra | 00:27 | |
| *** zz_dimtruck is now known as dimtruck | 00:29 | |
| jeblair | (an aside -- wrapping lines at at 78 characters makes it difficult to grep for error messages) | 00:30 |
| *** gildub has quit IRC | 00:31 | |
| nibalizer | is shade-launch-node something in infra or something in shade? | 00:32 |
| jeblair | nibalizer: something in infra; patch has not landed: https://review.openstack.org/234975 | 00:32 |
| jeblair | nibalizer: regardless, it's the only way to launch mirror nodes :| | 00:32 |
| jeblair | nibalizer: since (regular) launch-node really only works on rax and hpcloud | 00:33 |
| jeblair | nibalizer: so i'll probably switch back to that just to get this node up | 00:33 |
| jeblair | oh, except somehow novaclient got upgraded on the system and now that doesn't work anymore :( | 00:36 |
| *** Sukhdev has quit IRC | 00:39 | |
| nibalizer | jeblair: ugh | 00:39 |
| *** chenli has joined #openstack-infra | 00:40 | |
| *** oomichi has joined #openstack-infra | 00:40 | |
| *** chlong has quit IRC | 00:41 | |
| dstufft | jeblair: has this mirror not benes ynced in some time? | 00:41 |
| jeblair | dstufft: correct, it was months | 00:41 |
| dstufft | when we migrated to the new database server there was some blips where some pacakges got recorded in the DB but didn't get actually uploaded that I deleted | 00:41 |
| *** chlong has joined #openstack-infra | 00:42 | |
| dstufft | probably related ot that | 00:42 |
| jeblair | dstufft: ok, so my solution of removing them from todo was probably okay then | 00:42 |
| dstufft | yea | 00:42 |
| *** david-lyle has joined #openstack-infra | 00:42 | |
| jeblair | cool, seems to have been okay after that | 00:42 |
| dstufft | I can't wait till warehouse gets some saner transaction handling up on PyPI :/ | 00:43 |
| dstufft | part of the reason I'm going to be migrating to /packages/<sha256 hash>/<filename> is so we can write files to storage *before* committing the transaction without possibly overwriting a file | 00:44 |
| dstufft | (or at least, since the package name includes a hash of the file, if we do overwrite, it'll be with a file that is completely equal to what was already there) | 00:45 |
| jeblair | dstufft: ++; do you care about leaking those files in case the transaction doesn't complete? | 00:46 |
| jeblair | dstufft: (maybe clean them up periodically based on a check of the db?) | 00:46 |
| *** rbrndt_ has joined #openstack-infra | 00:47 | |
| dstufft | jeblair: not specifically, but I might just make a garbage collector job that looks for any files > some age that doesn't exist in the DB yea | 00:47 |
| jeblair | nibalizer: i downgraded novaclient and launched (regular) launch-node | 00:48 |
| *** chlong has quit IRC | 00:48 | |
| *** chlong has joined #openstack-infra | 00:49 | |
| *** ociuhandu has quit IRC | 00:49 | |
| *** zhurong has joined #openstack-infra | 00:51 | |
| *** EricGonczer_ has joined #openstack-infra | 00:52 | |
| *** chlong has quit IRC | 00:53 | |
| *** piet has quit IRC | 00:53 | |
| *** chlong has joined #openstack-infra | 00:54 | |
| *** FallenPegasus has joined #openstack-infra | 00:56 | |
| *** pvaneck_ has joined #openstack-infra | 00:57 | |
| nibalizer | jeblair: cool | 00:57 |
| *** ajmiller has quit IRC | 00:58 | |
| *** pvaneck_ has quit IRC | 01:01 | |
| *** pvaneck has quit IRC | 01:01 | |
| *** tphummel has quit IRC | 01:01 | |
| *** Daisy has joined #openstack-infra | 01:02 | |
| *** Qiming has joined #openstack-infra | 01:02 | |
| jeblair | hrm | 01:03 |
| jeblair | the kernel module failed to build on internap | 01:03 |
| jeblair | there's something funny going on with the kernel package there i think | 01:03 |
| *** Daisy has quit IRC | 01:03 | |
| *** amotoki has joined #openstack-infra | 01:04 | |
| jeblair | mgagne: where did linux version 3.16.0-59.79~14.04.1 come from? | 01:05 |
| jeblair | mgagne: it looks like a backported utopic kernel | 01:05 |
| *** zhurong has quit IRC | 01:06 | |
| *** ybathia has quit IRC | 01:08 | |
| mgagne | jeblair dpkg --list | grep kernel | 01:08 |
| *** sridhar_ram1 is now known as sridhar_ram | 01:08 | |
| prometheanfire | mordred: so... if I were to switch to default to be syslinux, that wouldn't mess anything up would it? | 01:08 |
| prometheanfire | just curious, probably not going to | 01:08 |
| prometheanfire | syslinux works just fine | 01:08 |
| jeblair | mgagne: http://paste.openstack.org/show/484941/ | 01:09 |
| prometheanfire | mordred: also, now doing minimal stage-4s https://gist.github.com/0ccef20b5dc82cbf5df5 | 01:09 |
| mgagne | lts-utopic =) | 01:09 |
| prometheanfire | building one now :D (from stage1, should take a while...) | 01:09 |
| mgagne | could be pull by cloud-initramfs-growroot | 01:09 |
| mgagne | or could also be the base image having lts installed by default | 01:09 |
| mgagne | lts-utopic | 01:10 |
| mgagne | https://wiki.ubuntu.com/Kernel/LTSEnablementStack | 01:10 |
| mgagne | jeblair ^ | 01:10 |
| *** yuanying has joined #openstack-infra | 01:10 | |
| mgagne | jeblair now it should be vivid, maybe it used to be utopic | 01:10 |
| *** peterlisak has quit IRC | 01:10 | |
| jeblair | mgagne: i kind of just want it to be trusty :) | 01:10 |
| *** kzaitsev_mb has quit IRC | 01:11 | |
| mgagne | jeblair I'm not sure how we can do that | 01:11 |
| mgagne | as the kernel is part of the base image AFAIK | 01:11 |
| jeblair | mgagne: i'm running trusty | 01:11 |
| *** yuanying_ has quit IRC | 01:11 | |
| *** onovy has quit IRC | 01:12 | |
| mgagne | jeblair yes, I know, I'm telling that Trusty 14.04.1 might be coming with lts-utopic | 01:12 |
| mgagne | and it's not something we do on purpose | 01:12 |
| mgagne | it's part of the Ubuntu release | 01:12 |
| nibalizer | jeblair: perhaps we could upload the ubuntu trusty image ourselves? either via dib or just the cannonical provided cloud images? | 01:12 |
| *** peterlisak has joined #openstack-infra | 01:12 | |
| jeblair | mgagne: hrm, that page makes it look like this is all add-on stuff; and it even recommends not doing it for cloud images... | 01:12 |
| mgagne | nibalizer I thought it was already the case, unless we are talking about a non-nodepool image | 01:12 |
| *** onovy has joined #openstack-infra | 01:12 | |
| jeblair | mgagne: maybe SpamapS can shed some light on it | 01:12 |
| jeblair | mgagne: we are talking about a non-nodepool image | 01:13 |
| mgagne | "If you have installed with older media you can use the following to install the newer kernel" | 01:13 |
| *** pc_m has quit IRC | 01:13 | |
| jeblair | mgagne: i'm spinning up a replacement mirror server, so using the internap trusty image | 01:13 |
| jeblair | mgagne: | 01:13 |
| nibalizer | I'm assuming some of jeblair's behaviour, but I think he's looking at the otput of 'nova image-list' and using the most trusty-looking ubuntu image in that list | 01:13 |
| jeblair | mgagne: (we don't have dib-built images for our long-lived servers yet, though we plan to, and it will solve this sort of problem) | 01:13 |
| jeblair | nibalizer, mgagne: correct, specifically "Ubuntu 14.04 LTS (Trusty Tahr)" | 01:14 |
| mgagne | we just can't ship a non-updated image, that would be negligence, like shipping windows without updates | 01:14 |
| jeblair | mgagne: i'm not asking you too | 01:14 |
| nibalizer | jeblair: perhaps http://apps.openstack.org/#tab=glance-images&asset=Ubuntu%20Trusty%2014 could be used in the interm between now and when we get dib-for-servers up? | 01:14 |
| jeblair | mgagne: i'm saying that somewhere, something has gone wrong and the trusty image is not a simple trusty image; it's trusty + stuff backported from new releases | 01:15 |
| *** pc_m has joined #openstack-infra | 01:15 | |
| *** Liuqing has joined #openstack-infra | 01:15 | |
| mgagne | let me reword: we won't ship an older version so you can get the "trusty" kernel as it's no longer part of what Ubuntu ships on their mirror. behavior is "expected" based on wiki page | 01:15 |
| jeblair | mgagne: i think you are misreading that | 01:15 |
| jeblair | mgagne: that page says "nd not recommended for cloud or virtual images" | 01:15 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue https://review.openstack.org/267323 | 01:16 |
| mgagne | jeblair well, I consider the "something went wrong" a misunderstanding or misinformation about what Ubuntu ships IRL | 01:16 |
| mgagne | and expectations are wrong | 01:16 |
| mgagne | jeblair I missed that part then =) | 01:16 |
| *** Daisy has joined #openstack-infra | 01:16 | |
| mgagne | jeblair we used this issue: http://releases.ubuntu.com/14.04/ubuntu-14.04.2-server-amd64.iso | 01:17 |
| *** alivigni has joined #openstack-infra | 01:17 | |
| *** salv-orlando has joined #openstack-infra | 01:17 | |
| jeblair | mgagne, nibalizer: so perhaps i should try manually uploading the cloud image nibalizer linked to at http://apps.openstack.org/#tab=glance-images&asset=Ubuntu%20Trusty%2014 and see if that has it | 01:19 |
| mgagne | proposed alternative solutions aren't great tbh: 1) install older versions (wth) 2) upgrade to newest LTS or downgrade to previous LTS (more WTH) 3) use netboot images instead of <release>-netboot images. (not sure what it means) | 01:19 |
| *** Swami has quit IRC | 01:19 | |
| nibalizer | mgagne: it is possible (and mostly supported) to plop a new kernel into ubuntu LTS, but ubuntu does maintain a trusty kernel for the entire lifetime of trusty | 01:19 |
| mgagne | jeblair last time we tested "vanilla" images, it had *very* outdated cloudinit and images wouldn't boot and configure itself. | 01:20 |
| jeblair | mgagne: yeesh. | 01:20 |
| jeblair | mgagne: if that fails, i'll try the 14.04.3 server image. | 01:20 |
| mgagne | jeblair you will have the vivid kernel | 01:20 |
| *** salv-orlando has quit IRC | 01:20 | |
| jeblair | mgagne: :( | 01:20 |
| alivigni | Question what is the stance on using Jenkins Job Builder vs. the Workflow plugin? | 01:20 |
| mgagne | jeblair unless you have a ubuntu cloud image | 01:20 |
| mgagne | jeblair but I lost all hopes in having an updated cloudinit | 01:21 |
| mgagne | jeblair which module are you trying to compile? AFS? | 01:21 |
| jeblair | mgagne: maybe i'll just perform surgery on this instance to unblock what i'm working on -- let's pick this up tomorrow with mordred and SpamapS | 01:21 |
| jeblair | mgagne: yep | 01:22 |
| mgagne | jeblair I guess you could forcefully downgrade the kernel | 01:22 |
| mgagne | but then I see it as a problem with the AFS kernel module as would it be a metal instance, you would have the same problem and "but it's cloud" wouldn't help you | 01:22 |
| *** gildub has joined #openstack-infra | 01:23 | |
| mgagne | jeblair what are the specific problem you have? is there a way I can test it on my side and see if upgrading to latest lts-vivid kernel fix it? | 01:23 |
| jeblair | mgagne: yes, there's a bug in the utopic kernel package for sure; but if i were running on metal, i would also remove the lts enablement stack unless needed :) | 01:23 |
| mgagne | jeblair as said, it's part of the base iso. little we can do about it, there is no cloud .iso | 01:24 |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Launch unit tests for only changed modules https://review.openstack.org/272346 | 01:24 |
| mgagne | so I don't know what I'm supposed to do. DIB might be the solution where we could download the ubuntu cloud image, crack it open, update cloud-init and ship it. | 01:25 |
| jeblair | mgagne: we'll get a lot more answers tomorrow with other folks around :) | 01:25 |
| mgagne | it's 8:30pm already :O | 01:26 |
| mgagne | EST* | 01:26 |
| *** erlon has quit IRC | 01:26 | |
| mgagne | let me know if you can find an option I can feed to preseed to not enable this kernel update | 01:27 |
| mgagne | jeblair does vivid kernel fix the problem? | 01:28 |
| * krotscheck reads scrollback | 01:28 | |
| mgagne | jeblair we could rebuild against 14.04.3 and see what happens | 01:28 |
| jeblair | mgagne: the _only_ kernel i want to run is trusty :) | 01:28 |
| mgagne | "it used to work before you touched the computer, put it back!" "yes mom..." =) | 01:29 |
| jeblair | mgagne: heh :) i have too many machines to have them be different | 01:29 |
| *** jpr has joined #openstack-infra | 01:29 | |
| krotscheck | jeblair:, mordred: Brief summary response to all the things, not necessarily in any order: Awesome! Wat. That's ironic. Well, it could be that it's adding a little extra lag because it's fetching things. | 01:30 |
| *** mtanino__ has quit IRC | 01:30 | |
| mgagne | jeblair https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64.manifest | 01:30 |
| mgagne | doesn't have lts kernel | 01:30 |
| mgagne | but our challenge is what our metal instances, we use the same image ^^' | 01:31 |
| *** apoorvad has quit IRC | 01:35 | |
| *** piet has joined #openstack-infra | 01:36 | |
| *** jpr has quit IRC | 01:36 | |
| *** ildikov has quit IRC | 01:36 | |
| *** angdraug has quit IRC | 01:37 | |
| *** sridhar_ram has quit IRC | 01:37 | |
| *** bpokorny has quit IRC | 01:38 | |
| *** r-daneel has joined #openstack-infra | 01:39 | |
| *** peterlisak has quit IRC | 01:39 | |
| *** r-daneel has quit IRC | 01:39 | |
| *** peterlisak has joined #openstack-infra | 01:39 | |
| jeblair | mgagne, nibalizer: utopectomy complete and http://mirror.nyj01.internap.openstack.org/ is up :) | 01:42 |
| mgagne | what happened? =) | 01:43 |
| *** sdake has quit IRC | 01:43 | |
| jeblair | mgagne: ? | 01:43 |
| mgagne | to the kernel | 01:43 |
| jeblair | mgagne: apt-get remove --purge | 01:43 |
| mgagne | =) | 01:43 |
| jeblair | mgagne: then reinstall of linux-image-generic | 01:43 |
| jeblair | mgagne: we rewrite sources.list, so we should't upgrade to it anymore | 01:44 |
| mgagne | you shouldn't be able to upgrade to lts-utopic afaik, it's a new package (not an upgrade) | 01:44 |
| jeblair | that too | 01:44 |
| jeblair | mgagne: and yeah, i make sure to uninstall the meta-packages too | 01:45 |
| *** FallenPegasus is now known as MarkAtwood_ | 01:45 | |
| *** flepied has quit IRC | 01:46 | |
| *** piet has quit IRC | 01:50 | |
| *** gildub has quit IRC | 01:50 | |
| jeblair | the bluebox mirror is up | 01:51 |
| *** jpr has joined #openstack-infra | 01:52 | |
| jeblair | (fwiw, if you set a 50G afs cache, the first time afs starts, it does take a bit as it pre-populates the cache dir) | 01:52 |
| *** salv-orlando has joined #openstack-infra | 01:54 | |
| *** piet has joined #openstack-infra | 01:54 | |
| *** sdake has joined #openstack-infra | 01:55 | |
| *** chlong has quit IRC | 01:55 | |
| *** chlong has joined #openstack-infra | 01:56 | |
| *** salv-orlando has quit IRC | 01:59 | |
| *** yuanying has quit IRC | 02:00 | |
| *** flepied has joined #openstack-infra | 02:00 | |
| *** amitgandhinz has joined #openstack-infra | 02:02 | |
| *** tiswanso has joined #openstack-infra | 02:02 | |
| nibalizer | utopectomy++ | 02:02 |
| *** jsavak has joined #openstack-infra | 02:03 | |
| *** yuanying has joined #openstack-infra | 02:03 | |
| jeblair | http://mirror.region-b.geo-1.hpcloud.openstack.org/ is up | 02:04 |
| jeblair | infra-root, krotscheck: the new afs-backed pypi mirrors in all of the nodepool cloud regions are up and running now | 02:05 |
| *** david-lyle has quit IRC | 02:05 | |
| jeblair | i'm rechecking https://review.openstack.org/272215 to spot any errors | 02:06 |
| jeblair | next step is https://review.openstack.org/238756 | 02:06 |
| *** tiswanso has quit IRC | 02:06 | |
| *** tiswanso has joined #openstack-infra | 02:07 | |
| *** dimtruck is now known as zz_dimtruck | 02:09 | |
| *** zhurong has joined #openstack-infra | 02:10 | |
| nibalizer | jeblair: great | 02:11 |
| *** davidlenwell has quit IRC | 02:14 | |
| *** yamamoto_ has joined #openstack-infra | 02:16 | |
| *** davidlenwell has joined #openstack-infra | 02:16 | |
| *** salv-orlando has joined #openstack-infra | 02:19 | |
| *** keedya has joined #openstack-infra | 02:19 | |
| *** fawadkhaliq has joined #openstack-infra | 02:21 | |
| *** baoli has joined #openstack-infra | 02:21 | |
| *** yuanying has quit IRC | 02:25 | |
| *** gongysh has joined #openstack-infra | 02:25 | |
| *** salv-orlando has quit IRC | 02:27 | |
| *** yuanying has joined #openstack-infra | 02:27 | |
| *** amitgandhinz has quit IRC | 02:28 | |
| *** rossella_s has quit IRC | 02:31 | |
| *** rossella_s has joined #openstack-infra | 02:32 | |
| jeblair | infra-root, krotscheck: it looks like we still have some _very_ out of date images, so we can't use the new mirrors until they are updated with the trusted hosts list | 02:32 |
| *** Daisy has quit IRC | 02:32 | |
| jpr | i'm seeing some problems during image create via nodepool on my openstackci deploy | 02:36 |
| jpr | I can get a simple image to build under a normal user account using `disk-image-create ubuntu vm` | 02:37 |
| *** fawadkhaliq has quit IRC | 02:38 | |
| jpr | if I register this definition under nodepool.yaml and i can get the build to start with nodepool only if I start the nodepoold. It then triggers a build with the command line "/bin/bash /usr/local/bin/disk-image-create -x -t --no-tmpfs -o /opt/nodepool_dib/5 ubuntu vm" (I'm concerned about the bare -t btw) | 02:39 |
| greghaynes | jpr: -t is for the output format | 02:39 |
| jpr | yes, is a bare -t ok? | 02:40 |
| greghaynes | jpr: it is not | 02:40 |
| greghaynes | jpr: you need to set | 02:40 |
| greghaynes | er, sorry | 02:40 |
| jpr | greghaynes: ok, so I'm assuming the only way to spec that is with the image-type in the provider section | 02:40 |
| greghaynes | jpr: yes | 02:41 |
| greghaynes | jpr: nodepool should handle that better and either fail or default to no -t | 02:41 |
| *** salv-orlando has joined #openstack-infra | 02:41 | |
| jpr | greghaynes: yeah. I thought that might be the case. i'll look at the code for that logic to see if there's something simple | 02:42 |
| *** yamamoto_ has quit IRC | 02:42 | |
| greghaynes | jpr: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/builder.py#n352 is likely useful ;) | 02:43 |
| jpr | in any case. the builds all fail with a bork of the ci node (ie. the main node in the openstackci config) because the /dev/ tree gets messed up. all the devices are deleted. don't know if this is from a bad bind mount or something intermediate that fails | 02:43 |
| greghaynes | jpr: oh crap | 02:43 |
| greghaynes | jpr: what elements? | 02:43 |
| greghaynes | diskimage-builder had a bug years ago that deleted all of /dev | 02:43 |
| *** Liuqing has quit IRC | 02:43 | |
| greghaynes | I hope we didnt repeat | 02:43 |
| jpr | greghaynes: definitely the ones for the dpc image but also the simplified ones I'm testing "ubuntu vm" | 02:44 |
| greghaynes | hrm | 02:44 |
| *** Liuqing has joined #openstack-infra | 02:44 | |
| greghaynes | I feel like I would have noticed that for ubuntu vm | 02:44 |
| jpr | gregheynes: if i run disk image build manually it works fine | 02:44 |
| greghaynes | ah | 02:44 |
| jpr | it's through nodepool that its a problem | 02:45 |
| greghaynes | that really should have no effect, so I dont have any good ideas | 02:45 |
| *** zhurong has quit IRC | 02:46 | |
| *** zhurong has joined #openstack-infra | 02:47 | |
| *** salv-orlando has quit IRC | 02:48 | |
| *** chlong has quit IRC | 02:49 | |
| jpr | greghaynes: btw, i can only initiate the nodepool image build via the cli if nodepoold itself is running. based on comments from asselin over the weekend, that may be expected behavior now | 02:49 |
| *** chlong has joined #openstack-infra | 02:50 | |
| greghaynes | jpr: it is. You could also run a nodepool-builder instead of nodepoold | 02:50 |
| *** ansiwen_ has joined #openstack-infra | 02:52 | |
| *** zhurong has quit IRC | 02:53 | |
| *** zhurong has joined #openstack-infra | 02:54 | |
| *** fawadkhaliq has joined #openstack-infra | 02:54 | |
| jpr | greghayes: I'll try setting image-type to see if that makes a difference in behavior | 02:55 |
| *** daemontool has quit IRC | 02:56 | |
| *** ansiwen has quit IRC | 02:59 | |
| jpr | greghaynes: here's the tail end of the image.log file during the nodepool image build. it looks like there's a mv command the moves all the entries from the bind-mounted dev dir into the new image.. That's likely the cause of the issue. things go south from there. http://paste.openstack.org/show/484948/ | 03:01 |
| *** pradk has quit IRC | 03:02 | |
| *** ansiwen_ has quit IRC | 03:03 | |
| *** ansiwen has joined #openstack-infra | 03:03 | |
| greghaynes | jpr: hrm, so can you file a bug and mention what distro youre building on, what elements you are using, and the command you ran? | 03:04 |
| greghaynes | jpr: its late here so I'll have to look at it tomorrow | 03:05 |
| greghaynes | jpr: a bug against diskimage-builder on launchpad, that is | 03:05 |
| jpr | greghaynes: yes. thanks for the target. also gonna try and debug a little further. | 03:05 |
| *** unicell has quit IRC | 03:06 | |
| *** tlian has quit IRC | 03:06 | |
| *** jsavak has quit IRC | 03:06 | |
| *** baoli has quit IRC | 03:08 | |
| *** gildub has joined #openstack-infra | 03:09 | |
| *** achanda has joined #openstack-infra | 03:09 | |
| *** alivigni has quit IRC | 03:10 | |
| *** zz_dimtruck is now known as dimtruck | 03:13 | |
| *** pradk has joined #openstack-infra | 03:16 | |
| *** chlong has quit IRC | 03:17 | |
| *** chlong has joined #openstack-infra | 03:18 | |
| *** chlong has quit IRC | 03:19 | |
| *** yuanying_ has joined #openstack-infra | 03:19 | |
| *** chlong has joined #openstack-infra | 03:20 | |
| *** yuanyin__ has joined #openstack-infra | 03:20 | |
| *** yuanying has quit IRC | 03:22 | |
| *** yuanying_ has quit IRC | 03:23 | |
| *** david-lyle has joined #openstack-infra | 03:25 | |
| *** baoli has joined #openstack-infra | 03:25 | |
| *** woodster_ has quit IRC | 03:26 | |
| *** yamamoto_ has joined #openstack-infra | 03:28 | |
| *** yuanyin__ has quit IRC | 03:29 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Configure GerritBot to Announce shovel changes https://review.openstack.org/272367 | 03:29 |
| *** yuanying has joined #openstack-infra | 03:29 | |
| *** Daisy has joined #openstack-infra | 03:33 | |
| *** unicell has joined #openstack-infra | 03:33 | |
| *** bnemec has quit IRC | 03:34 | |
| *** MarkAtwood_ has quit IRC | 03:34 | |
| *** rguillebert has quit IRC | 03:37 | |
| *** bnemec has joined #openstack-infra | 03:38 | |
| *** Daisy has quit IRC | 03:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:39 | |
| *** binbincong has quit IRC | 03:39 | |
| *** amotoki_ has joined #openstack-infra | 03:42 | |
| *** yamahata has quit IRC | 03:43 | |
| *** MarkAtwo_ has joined #openstack-infra | 03:44 | |
| *** sdake has quit IRC | 03:46 | |
| *** dims has joined #openstack-infra | 03:49 | |
| *** dims has quit IRC | 03:51 | |
| *** mtanino has quit IRC | 03:51 | |
| *** mtanino_ has quit IRC | 03:51 | |
| *** binbincong has joined #openstack-infra | 03:53 | |
| *** harlowja has quit IRC | 03:54 | |
| *** EricGonczer_ has quit IRC | 03:54 | |
| *** dims has joined #openstack-infra | 03:56 | |
| *** gongysh has quit IRC | 03:56 | |
| *** yuanying has quit IRC | 03:57 | |
| *** yuanying has joined #openstack-infra | 03:58 | |
| *** piet has quit IRC | 03:59 | |
| *** yuanying has quit IRC | 04:02 | |
| *** zhurong has quit IRC | 04:03 | |
| *** zhurong has joined #openstack-infra | 04:04 | |
| *** flwang1 has quit IRC | 04:04 | |
| *** chenli has quit IRC | 04:05 | |
| *** dims has quit IRC | 04:05 | |
| *** rguillebert has joined #openstack-infra | 04:05 | |
| *** wei-wang has joined #openstack-infra | 04:05 | |
| *** yuanying has joined #openstack-infra | 04:08 | |
| *** amotoki has quit IRC | 04:08 | |
| *** fawadkhaliq has quit IRC | 04:12 | |
| *** amotoki_ has quit IRC | 04:20 | |
| *** chenli has joined #openstack-infra | 04:20 | |
| *** achanda has quit IRC | 04:23 | |
| *** achanda has joined #openstack-infra | 04:23 | |
| *** diana_clarke has quit IRC | 04:26 | |
| *** amotoki has joined #openstack-infra | 04:28 | |
| *** vivekd has joined #openstack-infra | 04:28 | |
| *** nelsnelson has joined #openstack-infra | 04:29 | |
| *** rlandy has quit IRC | 04:29 | |
| *** nelsnelson has quit IRC | 04:31 | |
| *** nelsnelson has joined #openstack-infra | 04:31 | |
| *** fawadkhaliq has joined #openstack-infra | 04:32 | |
| *** wei-wang has quit IRC | 04:33 | |
| *** zhurong has quit IRC | 04:33 | |
| *** achanda has quit IRC | 04:36 | |
| *** piet has joined #openstack-infra | 04:38 | |
| *** dimtruck is now known as zz_dimtruck | 04:38 | |
| *** diana_clarke has joined #openstack-infra | 04:39 | |
| *** tiswanso has quit IRC | 04:44 | |
| *** stevemar has quit IRC | 04:47 | |
| *** vivekd has quit IRC | 04:48 | |
| *** jpr has quit IRC | 04:49 | |
| *** pradk has quit IRC | 04:56 | |
| *** stevemar_znc has joined #openstack-infra | 04:57 | |
| *** yamahata has joined #openstack-infra | 04:57 | |
| *** baoli has quit IRC | 04:58 | |
| *** piet has quit IRC | 04:59 | |
| *** cody-somerville has quit IRC | 05:00 | |
| *** ayoung is now known as ayoung_ZZZzzzz | 05:01 | |
| *** rbrndt_ has quit IRC | 05:02 | |
| *** piet has joined #openstack-infra | 05:03 | |
| *** zhurong has joined #openstack-infra | 05:07 | |
| *** pradk has joined #openstack-infra | 05:08 | |
| *** wei-wang has joined #openstack-infra | 05:13 | |
| *** changzhi has joined #openstack-infra | 05:13 | |
| *** ansiwen has quit IRC | 05:16 | |
| *** unicell1 has joined #openstack-infra | 05:21 | |
| *** Sukhdev has joined #openstack-infra | 05:21 | |
| *** unicell has quit IRC | 05:22 | |
| *** vivekd has joined #openstack-infra | 05:24 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 05:24 |
| *** yamamot__ has joined #openstack-infra | 05:25 | |
| *** yamamoto_ has quit IRC | 05:26 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr: Updated from global requirements https://review.openstack.org/272386 | 05:27 |
| *** abregman has joined #openstack-infra | 05:28 | |
| *** piet has quit IRC | 05:30 | |
| *** kzaitsev_mb has quit IRC | 05:31 | |
| *** jpr has joined #openstack-infra | 05:31 | |
| *** maishsk has quit IRC | 05:39 | |
| *** zhurong has quit IRC | 05:41 | |
| *** zhurong has joined #openstack-infra | 05:42 | |
| *** Liuqing has quit IRC | 05:44 | |
| *** Liuqing has joined #openstack-infra | 05:45 | |
| openstackgerrit | Wang Bo proposed openstack-dev/hacking: Add hacking check rule for not using xrange() https://review.openstack.org/272207 | 05:45 |
| *** hdd has quit IRC | 05:46 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Allow to omit list of projects in Gerrit trigger config https://review.openstack.org/272392 | 05:46 |
| *** baoli has joined #openstack-infra | 05:49 | |
| *** baoli has quit IRC | 05:54 | |
| *** ildikov has joined #openstack-infra | 06:02 | |
| *** mrmartin has joined #openstack-infra | 06:02 | |
| *** diana_clarke has quit IRC | 06:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/272399 | 06:11 |
| *** fawadkhaliq has quit IRC | 06:13 | |
| *** hichihara has quit IRC | 06:17 | |
| *** amotoki has quit IRC | 06:19 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Allow to omit list of projects in Gerrit trigger config https://review.openstack.org/272392 | 06:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/271902 | 06:23 |
| *** redrobot has left #openstack-infra | 06:26 | |
| *** redrobot has joined #openstack-infra | 06:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:27 | |
| *** achanda has joined #openstack-infra | 06:28 | |
| *** rossella_s has quit IRC | 06:31 | |
| *** rossella_s has joined #openstack-infra | 06:31 | |
| *** maishsk has joined #openstack-infra | 06:32 | |
| *** kzaitsev_mb has quit IRC | 06:32 | |
| *** pcrews has quit IRC | 06:35 | |
| openstackgerrit | Sergey Kraynev proposed openstack/requirements: Bump up heatclient version to 0.9.0 https://review.openstack.org/272405 | 06:36 |
| *** gyee has quit IRC | 06:37 | |
| *** rguillebert has quit IRC | 06:37 | |
| *** salv-orlando has joined #openstack-infra | 06:38 | |
| *** MarkAtwo_ has quit IRC | 06:38 | |
| *** mrmartin has quit IRC | 06:42 | |
| *** salv-orlando has quit IRC | 06:42 | |
| *** maishsk has quit IRC | 06:45 | |
| *** _nadya_ has joined #openstack-infra | 06:48 | |
| prometheanfire | circular dep? https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/openstack-repos/element-deps#L2 https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/puppet/element-deps#L2 | 06:49 |
| *** Sukhdev has quit IRC | 06:49 | |
| *** zhurong has quit IRC | 06:50 | |
| *** rcernin has joined #openstack-infra | 06:53 | |
| *** zhurong has joined #openstack-infra | 06:54 | |
| *** salv-orlando has joined #openstack-infra | 06:56 | |
| *** pcrews has joined #openstack-infra | 06:57 | |
| *** _nadya_ has quit IRC | 06:58 | |
| *** vivekd has quit IRC | 06:58 | |
| prometheanfire | also, that dep makes me think I'll need to support the puppt element | 06:58 |
| prometheanfire | mordred: ^ | 06:59 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/272399 | 07:04 |
| prometheanfire | also, looks like I'll need to add gentoo to bindep or somehow make a package mapping for gentoo | 07:05 |
| *** Daisy has joined #openstack-infra | 07:06 | |
| *** Daisy has quit IRC | 07:07 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Remove redundant tempest job for heatclient https://review.openstack.org/272411 | 07:07 |
| *** Daisy has joined #openstack-infra | 07:08 | |
| *** Daisy has quit IRC | 07:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Reduce bluebox max_servers from 39 to 0" https://review.openstack.org/272293 | 07:13 |
| *** binbincong has quit IRC | 07:14 | |
| AJaeger | ianw, yolanda: Could you review https://review.openstack.org/#/c/266565/ , please? It's a larger translation change that we need to move forward... | 07:14 |
| *** ybathia has joined #openstack-infra | 07:16 | |
| *** aopincar has joined #openstack-infra | 07:17 | |
| *** vgridnev has joined #openstack-infra | 07:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add job to test neutron against neutron-lib master https://review.openstack.org/268240 | 07:18 |
| *** wei-wang has quit IRC | 07:18 | |
| *** flepied has quit IRC | 07:19 | |
| *** cody-somerville has joined #openstack-infra | 07:19 | |
| *** fawadkhaliq has joined #openstack-infra | 07:20 | |
| *** Daisy has joined #openstack-infra | 07:22 | |
| *** [1]Thelo has joined #openstack-infra | 07:23 | |
| *** Thelo has quit IRC | 07:25 | |
| *** [1]Thelo is now known as Thelo | 07:25 | |
| *** binbincong has joined #openstack-infra | 07:26 | |
| dougwig | are the unit test log formats changing? i've never seen this kind of failure before: https://review.openstack.org/#/c/272413/ | 07:26 |
| *** nijaba has quit IRC | 07:27 | |
| AJaeger | dougwig: which failures? | 07:27 |
| *** amotoki has joined #openstack-infra | 07:27 | |
| *** nijaba has joined #openstack-infra | 07:27 | |
| *** scheuran has joined #openstack-infra | 07:27 | |
| AJaeger | dougwig: please wait 30 mins before a merge to project-config for zuul and jenkins to get updated... | 07:28 |
| dougwig | AJaeger: look at the unstable jobs in that link, and the scp failure at the end. | 07:28 |
| dougwig | AJaeger: oh right, that check experimental won't do anything. good call. | 07:28 |
| AJaeger | dougwig: no idea what's going on there ;( | 07:28 |
| *** simondodsley has quit IRC | 07:28 | |
| dougwig | AJaeger: that's straight up python-jobs template action there. | 07:29 |
| AJaeger | dougwig: we have quite a few networking changes in project-config, will you find some time to go through them and give a +1 in the next days? Should I add you or one of the other liasons to them? | 07:29 |
| AJaeger | dougwig: yeah, failure in scp ;( | 07:30 |
| dougwig | AJaeger: let me look now. and you're always welcome to add me. :) | 07:30 |
| *** sshnaidm has quit IRC | 07:30 | |
| AJaeger | dougwig: while it'S not ready, this one is also for you: https://review.openstack.org/#/c/271128/ | 07:32 |
| *** _nadya_ has joined #openstack-infra | 07:32 | |
| AJaeger | dougwig: thanks for reviewing! | 07:37 |
| dougwig | AJaeger: you beat me to correcting myself. :) | 07:43 |
| *** flepied has joined #openstack-infra | 07:44 | |
| *** ybathia has quit IRC | 07:45 | |
| AJaeger | ;) | 07:46 |
| AJaeger | 271128 needs some iterations and discussion... | 07:46 |
| *** salv-orlando has quit IRC | 07:46 | |
| *** esikachev has joined #openstack-infra | 07:47 | |
| dougwig | AJaeger: being up since 3am makes the -1 stick a bit heavy tonight. :) | 07:48 |
| AJaeger | dougwig: Oops, so take https://review.openstack.org/259536 https://review.openstack.org/270595 another time ;) | 07:49 |
| AJaeger | dougwig: I let you sleep now! | 07:49 |
| *** fawadkhaliq has quit IRC | 07:49 | |
| *** fawadkhaliq has joined #openstack-infra | 07:50 | |
| dougwig | AJaeger: oh wow, i think not a single +1 so far. | 07:56 |
| AJaeger | dougwig: https://review.openstack.org/#/c/259536/ has a depends on a *merged* https://review.openstack.org/#/c/244736/ | 07:57 |
| AJaeger | please check again... | 07:57 |
| dougwig | AJaeger: whoa, they merged a neutron stadium change when the PTL removed his +1? that doesn't look right. | 07:58 |
| AJaeger | dougwig: "I wish I didn't have to deal with these types of patches, and from now on, I am not going to. I let Kyle and/or Russell (or whoever is fond of the stadium concept and wants to to act as liaison) take over." | 07:59 |
| AJaeger | and kyle gave a +1 | 07:59 |
| AJaeger | dougwig: I'm fine waiting with 259536 if you want to discuss with armax. | 08:00 |
| dougwig | AJaeger: wow, ok, i don't think that's his current thinking, but that's what is in writing, so I won't hold that up while I clarify. | 08:00 |
| AJaeger | dougwig: let'S wait a few more days while you discuss... | 08:00 |
| *** jlanoux has joined #openstack-infra | 08:01 | |
| *** maishsk has joined #openstack-infra | 08:01 | |
| AJaeger | infra-root, we have lots of UNSTABLE jobs - errors out while uploading. Anybody around that could check this, please? | 08:02 |
| AJaeger | Most recent: https://jenkins07.openstack.org/job/gate-bifrost-python27/137/console | 08:02 |
| dougwig | AJaeger: pinged armax and ttx, and noted your correct observations. | 08:03 |
| yolanda | hi AJaeger, i can look | 08:03 |
| AJaeger | thanks, yolanda | 08:04 |
| AJaeger | thanks, dougwig ! | 08:04 |
| yolanda | mm, problems with cert.. | 08:04 |
| AJaeger | strange - https://jenkins03.openstack.org/job/gate-bifrost-docs/136/console succeeds and also uploaded... | 08:06 |
| *** MarkAtwo_ has joined #openstack-infra | 08:07 | |
| yolanda | mm, cert is not the problem, just a red herring, i looked at upper logs and there is a problem copying | 08:08 |
| yolanda | going to check static | 08:08 |
| AJaeger | yolanda: those run on rax - I was wondering whether my approval of https://review.openstack.org/272293 was premature and whether anything needed to be done manually there (not that I know of) but that's bluebox | 08:08 |
| *** arxcruz has joined #openstack-infra | 08:08 | |
| yolanda | AJaeger, that shouldn't cause problems, just force nodepool to stop providing nodes in bluebox | 08:09 |
| AJaeger | yolanda: bluebox is up again... | 08:10 |
| *** pahuang has quit IRC | 08:10 | |
| *** lezbar__ has quit IRC | 08:11 | |
| AJaeger | so, do we need to start nodepool there again? | 08:11 |
| dougwig | AJaeger: i had to go do a few non-neutron p-c reviews, to see if my +1 button still worked. | 08:11 |
| AJaeger | thanks, dougwig ! | 08:11 |
| yolanda | AJaeger, i cannot find anything strange on static, disk space is fine, and perms on that directories are ok | 08:12 |
| yolanda | do you have more samples of that failure? | 08:12 |
| *** cody-somerville has quit IRC | 08:14 | |
| dougwig | yolanda: https://review.openstack.org/#/c/272413/ | 08:14 |
| *** Daisy has quit IRC | 08:15 | |
| *** Daisy has joined #openstack-infra | 08:16 | |
| AJaeger | yolanda: Just check the red bullets on http://status.openstack.org/zuul/ | 08:16 |
| *** Hal has joined #openstack-infra | 08:16 | |
| yolanda | weird that it complains when copying, and about not finding the files, static has space and /srv/static/logs is present here | 08:16 |
| AJaeger | https://jenkins05.openstack.org/job/gate-ironic-python-agent-python27/278/ | 08:16 |
| *** armax has joined #openstack-infra | 08:16 | |
| *** Hal is now known as Guest44742 | 08:17 | |
| AJaeger | yolanda: yeah, and if you look at the bifrost jobs: 136 created the directory but 137 failed. | 08:17 |
| AJaeger | So, was there an update of software and suddenly the *second* mkdir fails of the same dir fails - while it worked before? | 08:17 |
| AJaeger | yolanda: a different problem - no puppet updates if puppetboard is correct: http://puppetboard.openstack.org/node/jenkins01.openstack.org | 08:19 |
| AJaeger | overview shows no results for 2+ hours | 08:20 |
| yolanda | oh dear, failures day :( | 08:20 |
| *** Daisy has quit IRC | 08:20 | |
| yolanda | i'm checking scp on jenkins first, trying to see if there is some problem, in terms of perms or keys | 08:20 |
| AJaeger | yolanda: sure, one issue at a time. I just wanted to mention it | 08:21 |
| *** dizquierdo has joined #openstack-infra | 08:21 | |
| *** flepied has quit IRC | 08:22 | |
| *** pahuang has joined #openstack-infra | 08:22 | |
| *** Daisy has joined #openstack-infra | 08:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:23 | |
| yolanda | weird, i test scp connection in jenkins, and looks fine | 08:23 |
| *** achanda has quit IRC | 08:23 | |
| *** _nadya__ has joined #openstack-infra | 08:24 | |
| *** Daisy has quit IRC | 08:24 | |
| *** shardy has joined #openstack-infra | 08:24 | |
| *** _nadya_ has quit IRC | 08:24 | |
| *** nmagnezi_ has joined #openstack-infra | 08:26 | |
| *** jcoufal has joined #openstack-infra | 08:26 | |
| yolanda | uh | 08:27 |
| yolanda | mkdir 272420 | 08:27 |
| yolanda | mkdir: cannot create directory `272420': Read-only file system | 08:27 |
| yolanda | looks as disk problems on static | 08:27 |
| *** mikelk has joined #openstack-infra | 08:28 | |
| *** amotoki has quit IRC | 08:28 | |
| yolanda | dmesg also shows a ton of errors on disk | 08:28 |
| AJaeger | yolanda: so, it fails for some jobs but works for others??? | 08:28 |
| AJaeger | Ooops ;( | 08:29 |
| AJaeger | time to send a service notice? | 08:29 |
| *** Daisy has joined #openstack-infra | 08:29 | |
| *** amotoki has joined #openstack-infra | 08:29 | |
| yolanda | AJaeger, please do it yes. I'll start a disk check | 08:30 |
| *** ifarkas has joined #openstack-infra | 08:30 | |
| AJaeger | yolanda: I have no permissions ;( You have to do it | 08:31 |
| yolanda | AJaeger sorry, i remember that i should have perms now yes, let me check manual and i'll do it | 08:31 |
| yolanda | #status alert Jenkins jobs are unstable and failing to upload artifacts, due to a disk problem on static server | 08:32 |
| yolanda | mm, that should have worked? | 08:33 |
| openstackgerrit | Ghanshyam Mann proposed openstack/requirements: Add yasfb (Sphinx extension) in global requirements https://review.openstack.org/270720 | 08:37 |
| AJaeger | yolanda: are you authenticated? | 08:37 |
| AJaeger | yolanda: I'll send an email to openstack-dev... | 08:39 |
| *** jaosorior has joined #openstack-infra | 08:40 | |
| yolanda | AJaeger, i am yes | 08:41 |
| *** Liuqing has quit IRC | 08:41 | |
| *** dtardivel has joined #openstack-infra | 08:41 | |
| AJaeger | strange - but the last of our worries | 08:41 |
| *** haiwei has joined #openstack-infra | 08:42 | |
| yolanda | maybe bot is down as well | 08:42 |
| *** Liuqing has joined #openstack-infra | 08:42 | |
| yolanda | i am running a check on the logs volume to try to repair | 08:42 |
| AJaeger | ok | 08:43 |
| *** kzaitsev_mb has quit IRC | 08:43 | |
| *** sshnaidm has joined #openstack-infra | 08:46 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Retire icehouse and juno for networking-midonet https://review.openstack.org/272005 | 08:47 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental rally job https://review.openstack.org/268859 | 08:47 |
| *** salv-orlando has joined #openstack-infra | 08:47 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental rally job https://review.openstack.org/268859 | 08:49 |
| yolanda | thanks for the email, volume is unmounted, and i'm running a filesystem check, it may take a while | 08:49 |
| *** armax has quit IRC | 08:49 | |
| *** esikachev has quit IRC | 08:51 | |
| electrical | long live hardware :-) | 08:52 |
| *** armax has joined #openstack-infra | 08:55 | |
| *** dizquierdo has quit IRC | 08:56 | |
| *** derekh has joined #openstack-infra | 08:56 | |
| *** fawadkhaliq has quit IRC | 08:57 | |
| *** fawadkhaliq has joined #openstack-infra | 08:57 | |
| *** BobBall_AWOL is now known as BobBall | 08:58 | |
| *** sshnaidm has quit IRC | 09:00 | |
| yolanda | (it's taking ages) | 09:00 |
| *** salv-orlando has quit IRC | 09:01 | |
| *** mriedem has joined #openstack-infra | 09:01 | |
| *** salv-orlando has joined #openstack-infra | 09:03 | |
| *** flepied has joined #openstack-infra | 09:05 | |
| *** hashar has joined #openstack-infra | 09:06 | |
| *** acabot has quit IRC | 09:06 | |
| *** esikachev has joined #openstack-infra | 09:07 | |
| *** hashar has quit IRC | 09:08 | |
| *** hashar has joined #openstack-infra | 09:09 | |
| *** lucas-dinner is now known as lucasagomes | 09:12 | |
| *** lezbar has joined #openstack-infra | 09:12 | |
| *** sshnaidm has joined #openstack-infra | 09:13 | |
| *** bexelbie has quit IRC | 09:15 | |
| *** bexelbie has joined #openstack-infra | 09:15 | |
| *** dtantsur|afk is now known as dtantsur | 09:16 | |
| *** _nadya__ has quit IRC | 09:18 | |
| *** ihrachys has joined #openstack-infra | 09:18 | |
| *** thorst has joined #openstack-infra | 09:18 | |
| *** jistr has joined #openstack-infra | 09:19 | |
| *** matrohon has joined #openstack-infra | 09:21 | |
| *** toabctl_ has quit IRC | 09:21 | |
| *** yaume has joined #openstack-infra | 09:22 | |
| *** Daisy has quit IRC | 09:22 | |
| anteaya | <- nova mid-cycle | 09:22 |
| *** toabctl_ has joined #openstack-infra | 09:22 | |
| *** markvoelker has quit IRC | 09:23 | |
| *** Daisy has joined #openstack-infra | 09:23 | |
| *** achanda has joined #openstack-infra | 09:24 | |
| yolanda | fsck in static arrived to step 2, checking directory structure... | 09:25 |
| *** jaypipes has joined #openstack-infra | 09:25 | |
| *** keedya has quit IRC | 09:26 | |
| *** ihrachys has quit IRC | 09:26 | |
| *** arxcruz has quit IRC | 09:27 | |
| *** electrofelix has joined #openstack-infra | 09:28 | |
| *** arxcruz has joined #openstack-infra | 09:28 | |
| mtreinish | yolanda, fungi: I'm trying to look at the logs for: https://review.openstack.org/#/c/270077/2 but the link for the layer4 tempest job is returning a not found | 09:29 |
| *** chenli has quit IRC | 09:29 | |
| yolanda | mtreinish, we are having problems with disk on static | 09:29 |
| mtreinish | http://logs.openstack.org/77/270077/2/check/gate-tempest-dsvm-layer4/6232886/ | 09:29 |
| mtreinish | yolanda: ah, ok. Thanks | 09:29 |
| yolanda | i had to unmount the volume and run fsck | 09:29 |
| *** kaisers has quit IRC | 09:29 | |
| yolanda | i tried to send an status update but not working for me, Andreas sent a post to openstack-dev mailing list | 09:29 |
| *** Liuqing has quit IRC | 09:29 | |
| *** sfinucan has joined #openstack-infra | 09:30 | |
| *** achanda has quit IRC | 09:30 | |
| *** Liuqing has joined #openstack-infra | 09:30 | |
| AJaeger | yolanda: if you have time for a longer review later on, please check 266565 | 09:32 |
| yolanda | ok | 09:33 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 09:34 |
| *** nihilifer has quit IRC | 09:35 | |
| *** kaisers has joined #openstack-infra | 09:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix zuul cloner macro for requirements repo https://review.openstack.org/271257 | 09:37 |
| *** Yanyanhu has joined #openstack-infra | 09:37 | |
| *** nihilifer has joined #openstack-infra | 09:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:38 | |
| esikachev | hi all! i need requirements cores for merge https://review.openstack.org/#/c/270676/ | 09:38 |
| AJaeger | esikachev: I haven't seen any of these around today besides the infra fallback | 09:39 |
| *** fawadkhaliq has quit IRC | 09:39 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: make spec-helper-beaker job voting https://review.openstack.org/272266 | 09:39 |
| *** fawadkhaliq has joined #openstack-infra | 09:39 | |
| esikachev | AJaeger: okay :( | 09:40 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Enable testr run passthrough arguments https://review.openstack.org/267824 | 09:40 |
| *** ociuhandu has joined #openstack-infra | 09:41 | |
| *** jordanP has joined #openstack-infra | 09:42 | |
| *** thorst has quit IRC | 09:44 | |
| *** unicell1 has quit IRC | 09:44 | |
| *** kzaitsev_mb has quit IRC | 09:44 | |
| *** thorst has joined #openstack-infra | 09:44 | |
| *** ociuhandu has quit IRC | 09:45 | |
| *** dizquierdo has joined #openstack-infra | 09:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new django extraction https://review.openstack.org/266565 | 09:47 |
| *** dtantsur is now known as dtantsur|brb | 09:48 | |
| *** thorst has quit IRC | 09:49 | |
| openstackgerrit | Merged openstack-infra/project-config: magnum-ui use generic django translation processing https://review.openstack.org/270596 | 09:49 |
| *** kzaitsev_mb has joined #openstack-infra | 09:50 | |
| *** Liuqing has quit IRC | 09:51 | |
| AJaeger | yolanda: if you took the volume off, why are jobs succeeding? "uploading file: '/srv/static/logs/73/270373/2/gate/gate-mistral-python27/1cc659b/testr_results.html.gz'" succeeded just a few minutes ago | 09:53 |
| yolanda | weird, volume is not mounted | 09:54 |
| *** hashar has quit IRC | 09:54 | |
| *** vivekd has joined #openstack-infra | 09:54 | |
| AJaeger | Ah, so it uploads content to the root disk... | 09:55 |
| yolanda | AJaeger, and that's uploading to static.openstack.org? | 09:55 |
| AJaeger | yes, see http://logs.openstack.org/73/270373/2/gate/gate-mistral-python27/1cc659b/console.html | 09:55 |
| AJaeger | so, you need to move content from root disk to safe place before mounting volume and then moving to volume... | 09:55 |
| yolanda | looks like... thanks for that warning. I see it's using root disk, yes | 09:56 |
| yolanda | fsck is on step 5 now | 09:56 |
| AJaeger | yolanda: do you want to enable nodepool for bluebox again? | 09:57 |
| yolanda | AJaeger, i am not aware of the causes to turn it off, do you know more about it? | 09:58 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 09:58 |
| AJaeger | yolanda: there was a maintenance window and they asked us to turn it off - and the window is open again... | 09:58 |
| yolanda | AJaeger, then i'm ok enabling it yes | 09:58 |
| *** Daisy has quit IRC | 10:04 | |
| *** Qiming has quit IRC | 10:05 | |
| *** salv-orl_ has joined #openstack-infra | 10:06 | |
| *** Daisy has joined #openstack-infra | 10:06 | |
| yolanda | AJaeger, is there some change we can revert? | 10:06 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Move Manila LVM Tempest job run in check queue https://review.openstack.org/272449 | 10:07 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Move Manila LVM Tempest job to check queue https://review.openstack.org/272449 | 10:08 |
| openstackgerrit | David Pursehouse proposed openstack-infra/lodgeit: Fix minor typo in README https://review.openstack.org/272450 | 10:08 |
| yolanda | AJaeger, fsck finished, so i'm moving logs and will mount after it's done | 10:08 |
| *** salv-orlando has quit IRC | 10:08 | |
| *** vivekd has quit IRC | 10:10 | |
| *** Daisy has quit IRC | 10:10 | |
| *** Daisy has joined #openstack-infra | 10:11 | |
| *** zhurong has quit IRC | 10:12 | |
| *** baoli has joined #openstack-infra | 10:13 | |
| *** Yanyanhu has quit IRC | 10:15 | |
| *** jamielennox is now known as jamielennox|away | 10:15 | |
| AJaeger | yolanda: don't know from reading backscroll. I thought initially merging the change would be fine... | 10:17 |
| AJaeger | glad to hear that fsck is finished | 10:17 |
| yolanda | i'm moving the old content now | 10:17 |
| mtreinish | yolanda: if you get a sec it looks like the health.o.o lost it's db connection: http://health.openstack.org/status if you could check the logs to see if there's an error | 10:17 |
| mtreinish | definitely a lower prio right now though | 10:17 |
| *** baoli has quit IRC | 10:18 | |
| yolanda | yep, let's see if volume is fine now and let's fix more things :) | 10:18 |
| yolanda | today is one of these days... | 10:18 |
| yolanda | AJaeger, move finished, and i can create directories now on that volume without errors | 10:19 |
| yolanda | let's see if jobs succeed | 10:19 |
| *** vivekd has joined #openstack-infra | 10:22 | |
| yolanda | mtreinish | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280792 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280809 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] return _ConnectionFairy._checkout(self) | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280818 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280833 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] fairy = _ConnectionRecord.checkout(pool) | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280844 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280859 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] rec = pool._do_get() | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280868 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280884 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] (self.size(), self.overflow(), self._timeout)) | 10:23 |
| yolanda | [Tue Jan 26 10:16:03.280912 2016] [:error] [pid 7181:tid 140191161534208] [remote 15.203.162.12:54483] TimeoutError: QueuePool limit of size 20 overflow 10 reached, connection timed out, timeout 30 | 10:23 |
| *** jaosorior has quit IRC | 10:23 | |
| *** markvoelker has joined #openstack-infra | 10:23 | |
| *** jaosorior has joined #openstack-infra | 10:24 | |
| *** MarkAtwo_ has quit IRC | 10:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out https://review.openstack.org/272331 | 10:24 |
| yolanda | mtreinish, i restarted apache there, is that better? | 10:25 |
| *** jamielennox|away is now known as jamielennox | 10:25 | |
| mtreinish | yolanda: yeah that fixed it | 10:25 |
| yolanda | ++ | 10:26 |
| mtreinish | but, I wonder what caused the queue pool to overflow | 10:26 |
| mtreinish | I can probably up that limit if it happens again | 10:26 |
| *** achanda has joined #openstack-infra | 10:27 | |
| *** yamamot__ has quit IRC | 10:27 | |
| *** markvoelker has quit IRC | 10:28 | |
| * AJaeger send an all clear by email.. | 10:29 | |
| *** Zara is now known as Zara__ | 10:31 | |
| *** rossella_s has quit IRC | 10:31 | |
| yolanda | going to check statusbot, it's weird that command didn't work for me, i should be admin | 10:31 |
| *** rossella_s has joined #openstack-infra | 10:31 | |
| *** Daisy has quit IRC | 10:32 | |
| dansmith | gerrit seems super slow again today.. is it time for a restart yet? | 10:33 |
| *** achanda has quit IRC | 10:33 | |
| AJaeger | Let's try #success | 10:33 |
| *** ldnunes has joined #openstack-infra | 10:33 | |
| AJaeger | #success Yolanda repaired static harddisk, so no more UNSTABLE jobs. | 10:34 |
| openstackstatus | AJaeger: Added success to Success page | 10:34 |
| yolanda | #success Volume on static is repaired | 10:34 |
| openstackstatus | yolanda: Added success to Success page | 10:34 |
| AJaeger | yolanda: so, that worked - it's the same bot but in this case it does not need permissions | 10:34 |
| yolanda | weird | 10:34 |
| yolanda | i may miss my creds somewhere else | 10:35 |
| AJaeger | ask jeblair or others later on for help | 10:36 |
| AJaeger | accessbot/channels.yaml looks fine, you're listed | 10:36 |
| AJaeger | yolanda: no new unstable jobs spotted so far | 10:36 |
| *** sshnaidm has quit IRC | 10:37 | |
| yolanda | nice | 10:37 |
| *** Zara has joined #openstack-infra | 10:37 | |
| yolanda | AJaeger, so what about bluebox? change alreay landed,so we should have bluebox with 39 nodes now? | 10:37 |
| AJaeger | we should - but we have not. | 10:38 |
| AJaeger | So, is nodepool stopped there? | 10:39 |
| yolanda | mm, looks as puppet not running | 10:39 |
| yolanda | because i still see max-servers to 0 on nodepool | 10:39 |
| openstackgerrit | Akihiro Motoki proposed openstack/requirements: Cap cliff-tablib <2.0 in stable/kilo https://review.openstack.org/272457 | 10:40 |
| *** fhubik has joined #openstack-infra | 10:40 | |
| AJaeger | yolanda: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-25.log.html#t2016-01-25T19:56:49 | 10:40 |
| yolanda | going to check puppetmaster | 10:40 |
| AJaeger | yolanda: otherwise let mordred and jeblair handle it later today... | 10:41 |
| yolanda | i need to run in a while, but i'm taking a look | 10:42 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out https://review.openstack.org/272331 | 10:43 |
| *** yamahata has quit IRC | 10:44 | |
| AJaeger | odyssey4me: do you have a governance change as well for 272331? | 10:44 |
| yolanda | last update i see for puppet-nodepool is 22 jan | 10:45 |
| *** salv-orl_ has quit IRC | 10:45 | |
| *** claudiub has joined #openstack-infra | 10:46 | |
| AJaeger | yolanda: see the link from eavesdrop and scroll up a few minutes - they mentioned somethign about puppet and nodepool there... | 10:46 |
| *** fhubik is now known as fhubik_brb | 10:47 | |
| AJaeger | so, might be disabled on purpose... | 10:47 |
| *** markmcd has quit IRC | 10:47 | |
| *** fhubik_brb is now known as fhubik | 10:48 | |
| *** ihrachys has joined #openstack-infra | 10:49 | |
| *** SotK is now known as SotK_ | 10:49 | |
| *** markmcd has joined #openstack-infra | 10:50 | |
| *** bmwiedemann has left #openstack-infra | 10:50 | |
| yolanda | AJaeger yes, it's on the list of disabled nodes | 10:51 |
| yolanda | i can apply the change manually | 10:51 |
| *** SotK has joined #openstack-infra | 10:51 | |
| *** sshnaidm has joined #openstack-infra | 10:51 | |
| *** changzhi has quit IRC | 10:52 | |
| *** _nadya_ has joined #openstack-infra | 10:54 | |
| yolanda | ok i updated max-servers manually on nodepool, it shall reload the config and start building on them | 10:55 |
| *** eliqiao has quit IRC | 10:55 | |
| *** dizquierdo has quit IRC | 10:56 | |
| *** Qiming has joined #openstack-infra | 10:56 | |
| *** rfolco has joined #openstack-infra | 10:56 | |
| AJaeger | thanks | 10:56 |
| *** thorst has joined #openstack-infra | 10:57 | |
| *** eliqiao has joined #openstack-infra | 10:57 | |
| yolanda | i need to go now, i'll be back in the afternoon | 10:59 |
| *** ansiwen has joined #openstack-infra | 10:59 | |
| *** dtantsur|brb is now known as dtantsur | 11:00 | |
| *** vivekd has quit IRC | 11:01 | |
| *** salv-orlando has joined #openstack-infra | 11:01 | |
| openstackgerrit | Ales Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs https://review.openstack.org/272469 | 11:02 |
| *** fhubik has quit IRC | 11:03 | |
| *** chlong has quit IRC | 11:05 | |
| *** kzaitsev_mb has quit IRC | 11:06 | |
| *** Qiming has quit IRC | 11:09 | |
| *** Qiming has joined #openstack-infra | 11:09 | |
| *** chlong has joined #openstack-infra | 11:10 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:12 | |
| *** thorst has quit IRC | 11:13 | |
| *** zeih has joined #openstack-infra | 11:15 | |
| *** thorst has joined #openstack-infra | 11:18 | |
| *** amotoki has quit IRC | 11:20 | |
| *** ihrachys has quit IRC | 11:21 | |
| *** MCoLo has quit IRC | 11:24 | |
| *** ihrachys has joined #openstack-infra | 11:24 | |
| *** baoli has joined #openstack-infra | 11:25 | |
| *** xwizard_ has joined #openstack-infra | 11:25 | |
| *** zhurong has joined #openstack-infra | 11:26 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out https://review.openstack.org/272331 | 11:26 |
| *** hashar has joined #openstack-infra | 11:26 | |
| *** zeih has quit IRC | 11:26 | |
| odyssey4me | AJaeger I've updated the review to include the link to the governance change, and added my vote. Thanks for the reminder. :) | 11:27 |
| *** claudiub has quit IRC | 11:27 | |
| *** erlon has joined #openstack-infra | 11:28 | |
| *** zeih has joined #openstack-infra | 11:28 | |
| *** kzaitsev_mb has quit IRC | 11:29 | |
| *** achanda has joined #openstack-infra | 11:30 | |
| AJaeger | thanks, yolanda ! Bluebox has nodes now.. | 11:31 |
| AJaeger | odyssey4me: thanks | 11:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector job timeouts https://review.openstack.org/272478 | 11:33 |
| *** zeih has quit IRC | 11:33 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 11:33 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client https://review.openstack.org/272479 | 11:34 |
| *** fawadkhaliq has quit IRC | 11:34 | |
| *** achanda has quit IRC | 11:35 | |
| *** rguillebert has joined #openstack-infra | 11:35 | |
| *** dizquierdo has joined #openstack-infra | 11:35 | |
| *** aysyd has joined #openstack-infra | 11:36 | |
| *** thorst has quit IRC | 11:37 | |
| *** zhurong has quit IRC | 11:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove timeout of ironic jobs https://review.openstack.org/272481 | 11:41 |
| *** zhurong has joined #openstack-infra | 11:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector job timeouts https://review.openstack.org/272478 | 11:43 |
| AJaeger | dtantsur: could you remove your -1, please? It survived the rebase... | 11:44 |
| *** vgridnev has quit IRC | 11:45 | |
| *** gildub has quit IRC | 11:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector and ironic-functional timeouts https://review.openstack.org/272478 | 11:56 |
| *** amotoki has joined #openstack-infra | 11:56 | |
| *** sdake has joined #openstack-infra | 11:58 | |
| dtantsur | AJaeger, +1 thanks for rebasing | 11:58 |
| AJaeger | thanks for reviewing, dtantsur | 11:59 |
| dtantsur | yolanda, hi! if you're by chance around, could you review/approve https://review.openstack.org/#/c/264168/ please? | 11:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Reduce timeout of ironic jobs https://review.openstack.org/272481 | 11:59 |
| dtantsur | AJaeger, btw, what's the reasoning behind reducing these timeouts? the ironic patch worries me, our gate is unstable enough already, and often suffers from tempest timeouts | 12:01 |
| *** mkoderer_cloud has joined #openstack-infra | 12:05 | |
| *** sdake_ has joined #openstack-infra | 12:09 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/irc-meetings: Revert to bi-weekly schedule for the neutron team meeting https://review.openstack.org/272494 | 12:09 |
| *** sdake has quit IRC | 12:11 | |
| *** salv-orlando has quit IRC | 12:11 | |
| *** jaosorior has quit IRC | 12:12 | |
| *** salv-orlando has joined #openstack-infra | 12:13 | |
| *** jaosorior has joined #openstack-infra | 12:13 | |
| *** sshnaidm has quit IRC | 12:13 | |
| *** daemontool has joined #openstack-infra | 12:14 | |
| *** sshnaidm has joined #openstack-infra | 12:14 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| *** baoli has quit IRC | 12:17 | |
| AJaeger | dtantsur: we use 60 mins as normal timeout, everything above is excpetion. Seems there'S a cargo-cult to copy the highest number and keep it without analysis ;( | 12:19 |
| AJaeger | dtantsur: if you really need longer, let's update again because the runtime is longer... | 12:19 |
| *** _amrith_ is now known as amrith | 12:20 | |
| AJaeger | dtantsur: if you have better numbers for the timeout, I'm happy to change but my look at quite a few recent ironic ones shows that it seldom is 50 mins and never more than 60 - so 70 mins should be fine. | 12:21 |
| *** doug-fish has joined #openstack-infra | 12:21 | |
| dtantsur | AJaeger, ok fine, let's wait for jroll to show up and review it | 12:21 |
| AJaeger | dtantsur: sure - that's why I added him ;) | 12:21 |
| *** vgridnev has joined #openstack-infra | 12:22 | |
| *** doug-fish has quit IRC | 12:23 | |
| *** yamamoto has quit IRC | 12:23 | |
| *** thorst has joined #openstack-infra | 12:24 | |
| *** doug-fish has joined #openstack-infra | 12:24 | |
| *** yamamoto has joined #openstack-infra | 12:24 | |
| *** markvoelker has joined #openstack-infra | 12:25 | |
| *** thorst has quit IRC | 12:28 | |
| *** thorst has joined #openstack-infra | 12:28 | |
| openstackgerrit | Ales Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs https://review.openstack.org/272469 | 12:28 |
| *** dims has joined #openstack-infra | 12:28 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** doug-fish has quit IRC | 12:28 | |
| *** markvoelker has quit IRC | 12:29 | |
| *** kushal has quit IRC | 12:30 | |
| *** doug-fish has joined #openstack-infra | 12:31 | |
| *** kushal has joined #openstack-infra | 12:32 | |
| *** kushal has quit IRC | 12:34 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 12:35 |
| *** weshay has quit IRC | 12:35 | |
| *** doug-fish has quit IRC | 12:35 | |
| *** vgridnev has quit IRC | 12:41 | |
| *** vgridnev has joined #openstack-infra | 12:42 | |
| *** Daisy has joined #openstack-infra | 12:43 | |
| *** dims has quit IRC | 12:45 | |
| *** Daisy has quit IRC | 12:48 | |
| *** rhallisey has joined #openstack-infra | 12:51 | |
| *** thorst_ has joined #openstack-infra | 12:52 | |
| *** pradk has quit IRC | 12:52 | |
| *** daemontool has quit IRC | 12:55 | |
| *** thorst has quit IRC | 12:55 | |
| *** salv-orlando has quit IRC | 12:57 | |
| *** baoli has joined #openstack-infra | 12:58 | |
| *** dprince has joined #openstack-infra | 12:59 | |
| *** zhurong has quit IRC | 13:01 | |
| *** baoli has quit IRC | 13:01 | |
| *** baoli has joined #openstack-infra | 13:01 | |
| *** zhurong has joined #openstack-infra | 13:01 | |
| *** fawadkhaliq has joined #openstack-infra | 13:02 | |
| *** bryan_att has joined #openstack-infra | 13:03 | |
| *** thorst_ has quit IRC | 13:03 | |
| lucasagomes | hi all, I've just started a new project for out-of-tree ironic drivers called ironic-staging-drivers the project has been approved already and I've put the initial commit up too (https://review.openstack.org/#/c/272499/) | 13:03 |
| openstackgerrit | Ales Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs https://review.openstack.org/272469 | 13:03 |
| lucasagomes | but I need core access to the project, someone can add me to the ironic-staging-drivers-{core, release} gerrit groups ? | 13:04 |
| *** trown|outttypeww is now known as trown | 13:04 | |
| lucasagomes | thank you | 13:04 |
| *** baoli has quit IRC | 13:05 | |
| lucasagomes | AJaeger, ^ is it something you can do? | 13:05 |
| *** thorst has joined #openstack-infra | 13:06 | |
| *** gongysh has joined #openstack-infra | 13:06 | |
| *** betherly has joined #openstack-infra | 13:07 | |
| * lucasagomes brb | 13:07 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Move the Neutron drivers meeting https://review.openstack.org/269780 | 13:07 |
| *** lucasagomes is now known as lucas-hungry | 13:07 | |
| betherly | AJaeger: hi there! | 13:07 |
| betherly | AJaeger: thanks again for your patch. having a look at it now | 13:07 |
| *** thorst has quit IRC | 13:07 | |
| betherly | so the project is not django although it does contain python. it is a plugin for horizon containing an angular based panel | 13:08 |
| *** thorst has joined #openstack-infra | 13:11 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: fix $LOG_DIR permissions https://review.openstack.org/272518 | 13:12 |
| *** adrienverge has joined #openstack-infra | 13:13 | |
| *** baoli has joined #openstack-infra | 13:14 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add reno jobs for puppet-keystone https://review.openstack.org/272261 | 13:14 |
| *** amrith is now known as _amrith_ | 13:16 | |
| *** baoli has quit IRC | 13:17 | |
| *** jsavak has joined #openstack-infra | 13:18 | |
| *** thorst_ has joined #openstack-infra | 13:18 | |
| *** zhurong has quit IRC | 13:18 | |
| *** gordc has joined #openstack-infra | 13:18 | |
| *** edmondsw has joined #openstack-infra | 13:18 | |
| *** weshay_xchat has joined #openstack-infra | 13:19 | |
| *** kgiusti has joined #openstack-infra | 13:19 | |
| *** gongysh has quit IRC | 13:21 | |
| *** thorst has quit IRC | 13:21 | |
| *** bcourt has quit IRC | 13:22 | |
| *** zz_dimtruck is now known as dimtruck | 13:22 | |
| *** baoli has joined #openstack-infra | 13:22 | |
| *** smarcet has joined #openstack-infra | 13:23 | |
| *** fawadkhaliq has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-infra | 13:25 | |
| *** weshay_xchat is now known as weshay | 13:27 | |
| *** markvoelker has quit IRC | 13:28 | |
| *** markvoelker has joined #openstack-infra | 13:28 | |
| *** ociuhandu has joined #openstack-infra | 13:29 | |
| *** dizquierdo has quit IRC | 13:29 | |
| AJaeger | lucas-hungry: enjoy lunch! I can't, you need one of the infra admins for that. | 13:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for Ironic bug 1537076 https://review.openstack.org/271496 | 13:29 |
| openstack | bug 1537076 in Ironic "Timed out waiting for Nova hypervisor-stats count >= 1 due to Nova Unable to establish connection to http://127.0.0.1:35357/v2.0/tokens" [High,Confirmed] https://launchpad.net/bugs/1537076 | 13:29 |
| *** doug-fish has joined #openstack-infra | 13:30 | |
| AJaeger | betherly: I need to see how to support this properly. If you follow the spec, we aim to give a framework that all repositories can follow for translation - special casing each and every one is complex and error prone. I'm not sure how angular behaves... | 13:31 |
| *** dimtruck is now known as zz_dimtruck | 13:32 | |
| *** achanda has joined #openstack-infra | 13:32 | |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting https://review.openstack.org/272538 | 13:37 |
| openstackgerrit | Ales Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs https://review.openstack.org/272469 | 13:37 |
| *** doug-fish has quit IRC | 13:37 | |
| *** achanda has quit IRC | 13:38 | |
| *** doug-fish has joined #openstack-infra | 13:38 | |
| *** piet has joined #openstack-infra | 13:38 | |
| *** burgerk has joined #openstack-infra | 13:38 | |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting https://review.openstack.org/272538 | 13:41 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: use SUDO to rename config/log files https://review.openstack.org/272518 | 13:42 |
| EmilienM | AJaeger: sorry, I updated it to use sudo ^ | 13:42 |
| *** doug-fish has quit IRC | 13:43 | |
| *** jaypipes has quit IRC | 13:44 | |
| AJaeger | EmilienM: ok | 13:44 |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting https://review.openstack.org/272538 | 13:44 |
| Kiall | akscram: dooh, missed your comment before PS#2 .. PS#3 uses python3-jobs | 13:44 |
| Kiall | ehh | 13:44 |
| Kiall | AJaeger: * | 13:44 |
| *** ldnunes_ has joined #openstack-infra | 13:45 | |
| betherly | AJaeger: which spec do i need to take a look at with regard the translation setup? i set the project up as per creators documentation but is there another doc im missing? | 13:46 |
| *** krtaylor has quit IRC | 13:46 | |
| betherly | having taken a look at the patch i think it will work correctly for the plugins needs but i will confirm | 13:46 |
| AJaeger | betherly: the one linked from the patch, it's http://specs.openstack.org/openstack-infra/infra-specs/specs/translation_setup.html | 13:46 |
| betherly | AJaeger: oh my bad! awesome thanks | 13:47 |
| *** terriyu has joined #openstack-infra | 13:47 | |
| AJaeger | betherly: the creators documentation did not fully cover translations, read the spec. | 13:47 |
| * AJaeger does not consider the spec cast in stone but the base for implementation, we'll update once we learn more - so comments are welcome | 13:47 | |
| AJaeger | betherly: for django you *need* django.pot as translation files - not sure whether angular is different... | 13:48 |
| *** ldnunes has quit IRC | 13:48 | |
| *** julim has joined #openstack-infra | 13:50 | |
| *** tlian has joined #openstack-infra | 13:52 | |
| esikachev | i need requirements cores for merge https://review.openstack.org/#/c/270676/ | 13:53 |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/elastic-recheck: Add query for Ironic bug 1537754 https://review.openstack.org/272541 | 13:53 |
| openstack | bug 1537754 in Ironic "Unable to associate floating IP 172.24.5.4 to fixed IP 10.1.0.5 for instance ..." [High,Triaged] https://launchpad.net/bugs/1537754 | 13:53 |
| *** asselin_ has joined #openstack-infra | 13:54 | |
| *** terriyu has quit IRC | 13:54 | |
| *** acabot has joined #openstack-infra | 13:54 | |
| *** jordanP has quit IRC | 13:55 | |
| devananda | mtreinish: another e-r query ^ | 13:55 |
| *** asselin__ has quit IRC | 13:56 | |
| *** piet has quit IRC | 13:57 | |
| pabelanger | morning | 13:58 |
| *** krtaylor has joined #openstack-infra | 13:58 | |
| *** asselin__ has joined #openstack-infra | 13:58 | |
| *** salv-orlando has joined #openstack-infra | 13:58 | |
| *** piet has joined #openstack-infra | 14:01 | |
| *** dkranz has joined #openstack-infra | 14:01 | |
| *** asselin_ has quit IRC | 14:01 | |
| *** asselin_ has joined #openstack-infra | 14:03 | |
| anteaya | morning pabelanger | 14:03 |
| *** salv-orlando has quit IRC | 14:03 | |
| *** dane-fichter_ has joined #openstack-infra | 14:04 | |
| rossella_s | hi Andreas I have a question for you. For Neutron I created a script to generate dynamically a file for gerrit-dash-creator to get a dashboard with critical bugs and patch related to approved blueprint. As said it's dynamic so the url for gerrit changes every time. | 14:04 |
| *** sriram has joined #openstack-infra | 14:04 | |
| rossella_s | AJaeger forgot to mention your nick in the msg before | 14:04 |
| dane-fichter_ | anteaya: hello | 14:05 |
| anteaya | dane-fichter_: hi | 14:05 |
| anteaya | I can comment on patches in gerrit: https://review.openstack.org/#/c/272292/2 | 14:05 |
| anteaya | this is the sandbox repo I just commented on this patch | 14:05 |
| rossella_s | AJaeger, going back to the question...On my server I update the link every day but I wanted to put the link in a public place. The Neutron doc is not an option because it would require the submission of a patch and to get it approved every time the link is updated. Any ideas? Thanks! | 14:05 |
| anteaya | have you been able to comment on patches in gerrit in the past, dane-fichter_ ? | 14:05 |
| dane-fichter_ | anteaya: yes. | 14:06 |
| *** lucas-hungry is now known as lucasagomes | 14:06 | |
| lucasagomes | AJaeger, thank you | 14:06 |
| anteaya | can you share the url of a patch you commented on dane-fichter_? | 14:06 |
| lucasagomes | I will look at someone that can add me | 14:06 |
| rossella_s | anteaya, hi! you might also have some ideas | 14:06 |
| *** asselin__ has quit IRC | 14:06 | |
| dane-fichter_ | anteaya: https://review.openstack.org/#/c/272071/ | 14:06 |
| *** asselin__ has joined #openstack-infra | 14:07 | |
| anteaya | rossella_s: actually I'm at nova mid-cycle and trying to concentrate on the discussion, sorry, just trying to help a nova dev troubleshoot their gerrit status | 14:07 |
| betherly | AJaeger: thats awesome. i will look into it further and let you know on it. are you happy for me to amend the patch? | 14:07 |
| rossella_s | anteaya, no worries, thanks for letting me know! | 14:07 |
| *** asselin_ has quit IRC | 14:08 | |
| dane-fichter_ | anteaya: I've been reviewing things for about a year now. For some reason today I am not being presented with the button that lets you review things | 14:08 |
| lucasagomes | hi all, I've just started a new project for out-of-tree ironic drivers called ironic-staging-drivers the project has been approved already and I've put the initial commit up too (https://review.openstack.org/#/c/272499/) | 14:08 |
| lucasagomes | but I need core access to the project, someone can add me to the ironic-staging-drivers-{core, release} gerrit groups ? | 14:08 |
| anteaya | dane-fichter_: you have two gerrit accounts | 14:08 |
| krotscheck | jeblair: Just got your note. Sounds like we just have to wait for the rest of the slaves to recycle? | 14:08 |
| *** amitgandhinz has joined #openstack-infra | 14:09 | |
| dane-fichter_ | anteaya: Ugh right. There was a weird capitalization bug. | 14:09 |
| *** rajinir has joined #openstack-infra | 14:09 | |
| *** amotoki has quit IRC | 14:09 | |
| anteaya | dane-fichter_: you do go to the gerrit text search box and enter: "reviewer:dane" you have two accounts, they ahve different email addresses | 14:09 |
| anteaya | dane-fichter_: what capitalization bug? | 14:09 |
| dane-fichter_ | anteaya: It had to do with stacklytics not tracking my reviews correctly with any caps in the e-mail address I think. It was over a year ago. | 14:10 |
| anteaya | dane-fichter_: my sense of things is that you have two gerrit accounts each with their own gerrit id | 14:11 |
| *** ansiwen has quit IRC | 14:11 | |
| *** jordanP has joined #openstack-infra | 14:11 | |
| dane-fichter_ | anteaya: Yeah I can see why that'd be a problem. Can we delete the unused one? | 14:11 |
| anteaya | dane-fichter_: no, you can't delete gerrit accounts, what we would have to do is figure out which one you want to keep and set the other one to disabled | 14:12 |
| *** ansiwen has joined #openstack-infra | 14:12 | |
| *** tiswanso has joined #openstack-infra | 14:13 | |
| anteaya | as well as acertain what your launchpad id is since if we don't access that gerrit will just create a new gerrit account if it things it is a new launchpad id | 14:13 |
| anteaya | dane-fichter_: fungi usually takes care of this issue of duplicate gerrit accounts but is away this week, perhaps once jeblair is online you can explain the situation and gently ask for some of his time to assist you | 14:14 |
| anteaya | this is low priority and we are on reduced infra-roots this week | 14:14 |
| anteaya | dane-fichter_: I counsel patience | 14:14 |
| *** rlandy has joined #openstack-infra | 14:14 | |
| anteaya | dane-fichter_: go into your gerrit settings, what is the gerrit id for the account you are signed into right now? | 14:15 |
| *** ansiwen has quit IRC | 14:16 | |
| anteaya | rossella_s: I hope you get some help for your thing, ping me in a few days if you still need assistance | 14:16 |
| dane-fichter_ | anteaya: dane-fichter | 14:16 |
| *** ansiwen has joined #openstack-infra | 14:16 | |
| anteaya | dane-fichter_: that is your username, gerrit id is a number | 14:16 |
| dane-fichter_ | anteaya: the account ID is 15524 | 14:16 |
| rossella_s | anteaya, thanks a lot! you are always very helpful. Enjoy the mid-cycle! | 14:16 |
| anteaya | dane-fichter_: thank you, stay in this channel, if jeblair has time he will ping you if he has more questions | 14:16 |
| anteaya | rossella_s: thanks for understanding :) | 14:17 |
| *** pblaho has quit IRC | 14:17 | |
| dane-fichter_ | anteaya: thanks! | 14:17 |
| anteaya | dane-fichter_: welcome | 14:17 |
| dane-fichter_ | anteaya: Seriously y'all are super helpful. Thanks. | 14:17 |
| anteaya | glad to help, thanks for the appreciation | 14:18 |
| anteaya | very grateful | 14:18 |
| anteaya | :) | 14:18 |
| *** pblaho has joined #openstack-infra | 14:18 | |
| anteaya | jeblair: priority, graphite seems to have stopped reporting | 14:19 |
| anteaya | infra-root ^^ | 14:19 |
| *** sabeen1 has joined #openstack-infra | 14:20 | |
| *** bhunter71 has left #openstack-infra | 14:21 | |
| anteaya | did graphite perhaps get adversly affected when yolanda dealt with the logging issue of a few hours ago? | 14:23 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Add possibility to use SSH instead of HTTP to get ref list https://review.openstack.org/246433 | 14:23 |
| AJaeger | betherly: sure, take over my change - and tell me about anything you find. Thanks! | 14:24 |
| *** daemontool has joined #openstack-infra | 14:24 | |
| AJaeger | rossella_s: I'M here now.. | 14:24 |
| betherly | AJaeger: thanks and will do | 14:25 |
| *** doug-fish has joined #openstack-infra | 14:26 | |
| AJaeger | rossella_s: what'S the content of the script? How do you update it? Manually add bugs etc? | 14:26 |
| *** doug-fish has quit IRC | 14:26 | |
| *** dims has joined #openstack-infra | 14:27 | |
| rossella_s | AJaeger, this is the script https://github.com/openstack/neutron/blob/master/tools/milestone-review-dash.py | 14:27 |
| *** doug-fish has joined #openstack-infra | 14:27 | |
| rossella_s | AJaeger, you just run it passing the milestone you want to track, then it writes a .dash file that can be used by gerrit-dash-creator to produce the url | 14:28 |
| *** sabeen3 has joined #openstack-infra | 14:28 | |
| AJaeger | so, this searches bugs in launchpad and generates the review from it? | 14:28 |
| *** peristeri has joined #openstack-infra | 14:29 | |
| rossella_s | AJaeger, yes | 14:29 |
| *** dims has quit IRC | 14:29 | |
| *** sabeen1 has quit IRC | 14:30 | |
| rossella_s | AJaeger, I have a cron job right now on my server that generates the url daily and publish it on my blog. I want to publish it in a public place...without having to create a gerrit review every day :) | 14:30 |
| AJaeger | several options in general - but I'm not sure which one is best, this would be for the broader team to discus: | 14:30 |
| AJaeger | a periodic job (that's like cron) that publishes a document that includes the URL | 14:30 |
| rossella_s | AJaeger, but then somebody should approve it right? | 14:31 |
| *** rossella_s has quit IRC | 14:31 | |
| *** abregman is now known as abregman|mtg | 14:31 | |
| *** tiswanso has quit IRC | 14:31 | |
| *** rossella_s has joined #openstack-infra | 14:31 | |
| AJaeger | we normally publish docs from the post job whenever something changes - so that would mean a change | 14:32 |
| *** tiswanso has joined #openstack-infra | 14:32 | |
| AJaeger | But a periodic queue could do it as well - but we have no such job, so that's why it needs broader discussion. | 14:32 |
| *** dims has joined #openstack-infra | 14:32 | |
| AJaeger | Since most infra folks are travelling this week, what about sending an email to openstack-infra? | 14:33 |
| rossella_s | AJaeger, ok, thanks for your help! | 14:33 |
| *** mkoderer_cloud has quit IRC | 14:34 | |
| AJaeger | you're welcome - and if you need help further on, please ask | 14:34 |
| *** pradk has joined #openstack-infra | 14:34 | |
| *** rajinir has quit IRC | 14:34 | |
| *** boris-42 has quit IRC | 14:34 | |
| *** roaet_ has quit IRC | 14:35 | |
| *** iremizov has quit IRC | 14:35 | |
| *** pblaho has quit IRC | 14:35 | |
| *** boris-42 has joined #openstack-infra | 14:35 | |
| *** davidlenwell has quit IRC | 14:35 | |
| *** rajinir has joined #openstack-infra | 14:36 | |
| *** iremizov has joined #openstack-infra | 14:36 | |
| *** mkoderer_cloud has joined #openstack-infra | 14:36 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/271902 | 14:37 |
| AJaeger | dims: no sense in rechecking the argparse removal yet - still too many repos have it ;( | 14:37 |
| *** ansiwen has quit IRC | 14:37 | |
| dims | AJaeger : ack :( just hoping against hope | 14:38 |
| *** ansiwen has joined #openstack-infra | 14:38 | |
| AJaeger | dims: check https://review.openstack.org/#/q/status:open++branch:master+topic:rm-argparse - it still contains several python clients and dib which is blocked by ironic... - once that list does not contain entries from projects.txt, you should be fine AFAIK | 14:38 |
| *** ansiwen has quit IRC | 14:39 | |
| *** dmsimard is now known as dmsimard|afk | 14:39 | |
| *** pblaho has joined #openstack-infra | 14:39 | |
| *** roaet_ has joined #openstack-infra | 14:41 | |
| *** zz_dimtruck is now known as dimtruck | 14:42 | |
| *** _amrith_ is now known as amrith | 14:43 | |
| *** daemontool has quit IRC | 14:43 | |
| *** coreycb` is now known as coreycb | 14:44 | |
| *** daemontool has joined #openstack-infra | 14:47 | |
| *** asselin_ has joined #openstack-infra | 14:47 | |
| AJaeger | dims, feel free to ping folks ;) | 14:47 |
| *** asselin__ has quit IRC | 14:48 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 14:49 |
| *** fawadkhaliq has joined #openstack-infra | 14:49 | |
| *** jlanoux has quit IRC | 14:49 | |
| *** davidlenwell has joined #openstack-infra | 14:49 | |
| *** jlanoux has joined #openstack-infra | 14:51 | |
| jlk | jeblair: mordred: how has the Blue Box nodepool been performing today? | 14:51 |
| anteaya | jlk: can't tell, graphite is down | 14:52 |
| jlk | so I noticed | 14:52 |
| anteaya | there seemed to be a traffic spike for graphite and it isn't reporting now | 14:52 |
| anteaya | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=25&page=2 | 14:52 |
| *** Qiming has quit IRC | 14:53 | |
| *** jpr has quit IRC | 14:53 | |
| anteaya | jlk: prior to that it looks like bluebox had a spike (which had errors) but other than that looked error free | 14:53 |
| jlk | okay. Mostly I was curious if build times and run times got "faster" perhaps. The change out for underlying instance storage was kind of a big one | 14:54 |
| anteaya | jlk: I understand | 14:54 |
| *** regXboi has joined #openstack-infra | 14:54 | |
| *** salv-orlando has joined #openstack-infra | 14:54 | |
| anteaya | and would like to get you some data | 14:54 |
| *** jaypipes has joined #openstack-infra | 14:55 | |
| anteaya | unfortunately I have no data to share | 14:55 |
| anteaya | I'm sorry | 14:55 |
| anteaya | :( | 14:55 |
| jlk | heh, no worries | 14:55 |
| jlk | it's not urgent | 14:55 |
| anteaya | jlk: hopefully when jeblair is online and after he gets graphite up again | 14:56 |
| anteaya | perhaps he might have some numbers | 14:56 |
| anteaya | jlk: thanks for asking | 14:56 |
| anteaya | and thanks for doing the work, woooo! | 14:56 |
| *** salv-orlando has quit IRC | 14:57 | |
| *** jlanoux has quit IRC | 14:57 | |
| *** jlanoux has joined #openstack-infra | 14:58 | |
| *** ansiwen has joined #openstack-infra | 14:58 | |
| *** piet has quit IRC | 14:59 | |
| *** angdraug has joined #openstack-infra | 15:00 | |
| *** ilyashakhat has joined #openstack-infra | 15:00 | |
| *** placko_ has joined #openstack-infra | 15:01 | |
| *** dizquierdo has joined #openstack-infra | 15:01 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Dismiss glance-drivers team https://review.openstack.org/272573 | 15:01 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for Ironic bug 1537754 https://review.openstack.org/272541 | 15:01 |
| openstack | bug 1537754 in Ironic "Unable to associate floating IP 172.24.5.4 to fixed IP 10.1.0.5 for instance ..." [High,Triaged] https://launchpad.net/bugs/1537754 | 15:01 |
| *** rubasov has joined #openstack-infra | 15:01 | |
| *** alivigni has joined #openstack-infra | 15:02 | |
| *** pblaho has quit IRC | 15:03 | |
| *** EricGonczer_ has joined #openstack-infra | 15:04 | |
| *** EricGonc_ has joined #openstack-infra | 15:06 | |
| *** jlanoux has quit IRC | 15:06 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 15:07 |
| *** amotoki has joined #openstack-infra | 15:08 | |
| *** EricGonczer_ has quit IRC | 15:09 | |
| *** jlanoux has joined #openstack-infra | 15:09 | |
| *** salv-orlando has joined #openstack-infra | 15:10 | |
| rubasov | hello there | 15:10 |
| rubasov | did I come to the proper channel if I have a gerrit nuisance? | 15:11 |
| *** salv-orlando has quit IRC | 15:11 | |
| anteaya | what is your situation? | 15:11 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:11 | |
| anteaya | gerrit questions are answered in this channel, yes | 15:11 |
| rubasov | my user can't be added to reviews | 15:11 |
| *** salv-orlando has joined #openstack-infra | 15:11 | |
| rubasov | like this | 15:11 |
| rubasov | Bence Romsics <bence.romsics@ericsson.com> does not identify a registered user or group | 15:11 |
| rubasov | I have found this bug | 15:11 |
| rubasov | https://bugs.launchpad.net/openstack-ci/+bug/1394191 | 15:11 |
| openstack | Launchpad bug 1394191 in OpenStack Core Infrastructure "can't be added as a gerrit reviewer " [Medium,Fix released] - Assigned to Jeremy Stanley (fungi) | 15:11 |
| rubasov | sounds like just my problem | 15:12 |
| anteaya | rubasov: you have two accounts | 15:12 |
| *** esker has quit IRC | 15:12 | |
| rubasov | yes, I don't use one of them | 15:12 |
| rubasov | can I delete one? | 15:12 |
| anteaya | As discussed in #openstack-infra on IRC, this was because you tried authenticating with a second Launchpad OpenID account, causing Gerrit to generate conflicting accounts for the same E-mail address. At your request I have disabled the second account in Gerrit. | 15:12 |
| *** esker has joined #openstack-infra | 15:13 | |
| anteaya | no, an infra-root with gerrit permissions has to do that | 15:13 |
| *** piet has joined #openstack-infra | 15:13 | |
| anteaya | rubasov: what is the gerrit id from your settings for the account you are currently in? | 15:13 |
| rubasov | a moment | 15:13 |
| *** esker has quit IRC | 15:14 | |
| rubasov | this is the one I'm actually using: | 15:14 |
| *** esker has joined #openstack-infra | 15:14 | |
| rubasov | Account ID: 15554 | 15:14 |
| anteaya | thank you | 15:14 |
| anteaya | so this is the account you wish to keep, the one you are in now? | 15:14 |
| rubasov | exactly | 15:14 |
| *** tiswanso has quit IRC | 15:15 | |
| *** jlanoux has quit IRC | 15:15 | |
| anteaya | thank you, when a gerrit admin is online hopefully they will read this information in backscroll and hopefully have time to help | 15:15 |
| anteaya | this issue is low priority, so stay in channel and have patience | 15:15 |
| *** tiswanso has joined #openstack-infra | 15:15 | |
| *** tiswanso has quit IRC | 15:15 | |
| rubasov | anteaya: thanks a lot for your help | 15:15 |
| rubasov | will stay and wait | 15:15 |
| anteaya | thanks in advance for your patience | 15:16 |
| anteaya | thank you | 15:16 |
| rubasov | thank you, have a nice day | 15:16 |
| anteaya | highest priorty right now is getting graphite reporting again | 15:16 |
| anteaya | rubasov: and you | 15:16 |
| *** jlanoux has joined #openstack-infra | 15:16 | |
| pabelanger | any project-config-core mind helping 264994 land? Adds templating support to one of our dashboards | 15:18 |
| yolanda | hi, back... anteaya, is there still the problem with graphite? | 15:19 |
| anteaya | yes | 15:19 |
| krotscheck | mordred: You around? Do you know what the command is for "Execute things with these afs credentials?" | 15:19 |
| anteaya | yolanda: thanks, can you take a look at graphite? it isn't reporting | 15:19 |
| * krotscheck is patch digging | 15:19 | |
| anteaya | yolanda: it looks like it had a traffic spike and then the service stopped shortly thereafter | 15:20 |
| anteaya | yolanda: I'm not sure if it is related to the issue you had with static.o.o or not | 15:20 |
| krotscheck | mordred: Ah, nevermind, found it | 15:20 |
| yolanda | anteaya, i don't think it could be related, as the problem with static was a disk corruption on a volume, but i'll look at graphite logs | 15:21 |
| anteaya | yolanda: thanks | 15:21 |
| lucasagomes | yolanda, mordred you are infra-admin right? Do you mind adding myself to the ironic-staging-drivers-core gerrit group? The ironic-staging-project got approved already but the core gerrit group is empty | 15:21 |
| lucasagomes | so I can't approve anything there :-) | 15:21 |
| *** mtanino has joined #openstack-infra | 15:21 | |
| *** nmagnezi_ has quit IRC | 15:22 | |
| *** mtanino_ has joined #openstack-infra | 15:22 | |
| *** zhurong has joined #openstack-infra | 15:22 | |
| *** esker has quit IRC | 15:22 | |
| *** mtanino__ has joined #openstack-infra | 15:23 | |
| *** ajmiller has joined #openstack-infra | 15:24 | |
| *** pblaho has joined #openstack-infra | 15:24 | |
| *** zhurong has quit IRC | 15:25 | |
| *** rbrndt_ has joined #openstack-infra | 15:26 | |
| *** david-lyle_ has joined #openstack-infra | 15:26 | |
| *** simondodsley has joined #openstack-infra | 15:26 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Allow using numeric change number in addition to ID in Depends-On tag https://review.openstack.org/215553 | 15:27 |
| *** tonytan4ever has joined #openstack-infra | 15:28 | |
| *** burgerk has quit IRC | 15:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Allow builder.py to start without gearman servers https://review.openstack.org/272590 | 15:29 |
| pabelanger | greghaynes: clarkb: ^ minor change to nodepool-builder | 15:29 |
| anteaya | pabelanger: clarkb is away for the week | 15:29 |
| pabelanger | anteaya: thanks | 15:29 |
| anteaya | welcome | 15:29 |
| *** yamahata has joined #openstack-infra | 15:30 | |
| *** tiswanso has joined #openstack-infra | 15:30 | |
| *** MarkAtwo_ has joined #openstack-infra | 15:30 | |
| *** jlanoux has quit IRC | 15:30 | |
| yolanda | anteaya, i restarted carbon-cache and statsd, and restarted apache as well. Let's see if that helps | 15:31 |
| yolanda | oh dear, disk corruption also there! | 15:32 |
| anteaya | yolanda: did the logs show any useful hints about why it quit? | 15:32 |
| yolanda | exceptions.IOError: [Errno 30] Read-only file system: '/var/lib/graphite/storage/whisper/stats/haproxy/balance_git_daemon/git06/openstack/org/eresp.wsp' | 15:32 |
| anteaya | oh no :( | 15:32 |
| yolanda | maybe rax had some power outage? | 15:32 |
| *** jlanoux has joined #openstack-infra | 15:32 | |
| anteaya | yolanda: can you pm the server uuids to mikal please? | 15:33 |
| mordred | yolanda: I do not know enough about recovering from that to be useful | 15:33 |
| yolanda | going to unmount the volume and proceed with an fsck as i did on static | 15:33 |
| mordred | cool | 15:33 |
| anteaya | mordred: morning | 15:33 |
| mordred | yolanda: did we have some disk corruption? | 15:33 |
| openstackgerrit | Merged openstack-infra/project-config: Switch to IPA as a primary ramdisk for ironic-inspector jobs https://review.openstack.org/264168 | 15:33 |
| yolanda | mordred, we had on static this morning, on /srv/static/logs | 15:33 |
| yolanda | and now i'm seeing the same on graphite | 15:33 |
| mordred | awesome | 15:33 |
| mordred | that's fantastic | 15:34 |
| anteaya | yolanda: mikal is with rax and sitting here beside me at the nova mid-cycle, he will try to get some information on our servers if he has the uuids | 15:34 |
| *** isaacb has joined #openstack-infra | 15:34 | |
| mordred | yolanda: I'm mostly cheering from the side - but I can get uuids to mikal from here | 15:34 |
| *** salv-orlando has quit IRC | 15:34 | |
| *** nmagnezi_ has joined #openstack-infra | 15:34 | |
| yolanda | thanks mordred | 15:34 |
| *** placko_ has quit IRC | 15:34 | |
| anteaya | I'm not sure if that should be private or not so defaulting to private | 15:34 |
| anteaya | apologies if I'm wrong | 15:34 |
| yolanda | i'll work on the volume fix | 15:35 |
| anteaya | mordred: awesome, thank you | 15:35 |
| *** salv-orlando has joined #openstack-infra | 15:35 | |
| mikal | Is the disk corruption on volumes or ephemeral disk? | 15:35 |
| mordred | volumes | 15:35 |
| mordred | wait | 15:35 |
| mordred | I might be lying | 15:35 |
| mikal | Ok, so volume ids as well would be good then | 15:35 |
| mordred | but I'm pretty sure volumes | 15:35 |
| *** sridhar_ram has joined #openstack-infra | 15:35 | |
| *** abregman|mtg is now known as abregman | 15:35 | |
| *** achanda has joined #openstack-infra | 15:36 | |
| *** ajmiller has quit IRC | 15:37 | |
| *** sdake_ is now known as sdake | 15:38 | |
| jroll | mordred: is this dfw | 15:38 |
| *** ajmiller has joined #openstack-infra | 15:38 | |
| yolanda | ok, fsck running on graphite | 15:38 |
| jroll | and volumes | 15:38 |
| jroll | ? | 15:38 |
| yolanda | mordred, can you send some status notice? i tried this morning but for some reason i still don't have the powers | 15:38 |
| *** jpr has joined #openstack-infra | 15:38 | |
| jroll | mordred: if so, https://status.rackspace.com/index/viewincidents?group=21&start=1453784400 could have cut iscsi connections | 15:38 |
| jroll | mikal: ^ | 15:39 |
| anteaya | mordred: please send an email to mikal and johnthetubaguy including a description of the issue on the static.o.o server and the graphite server including server uuids and cinder volume ids | 15:39 |
| mordred | anteaya: I have private messaged mikal uuids | 15:39 |
| anteaya | mordred: they will find someone to forward that email to in order to escalate | 15:39 |
| mordred | but I will also send an email | 15:39 |
| jroll | anteaya: ^ see that link if this is with cinder volumes in dfw | 15:39 |
| jroll | could be related | 15:39 |
| mordred | jroll: it is | 15:40 |
| anteaya | mordred: thank you, that is the latest request | 15:40 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 15:40 |
| mordred | anteaya: I think we're likely fine once yolanda fscks things - although we might need to do the other servers with volumes too, depending on how they are | 15:40 |
| anteaya | jroll: thank you | 15:40 |
| anteaya | mordred: very good | 15:40 |
| anteaya | mikal is eating a piece of cake in support of the work to fix things | 15:41 |
| *** achanda has quit IRC | 15:41 | |
| anteaya | go yolanda | 15:41 |
| * anteaya also will wander to the dessert table | 15:41 | |
| mordred | elasticsearch, afs01.dfw, mirror.dfw, and nodpool all also have volumes | 15:41 |
| yolanda | yep, fsck running but it will be slow | 15:41 |
| mordred | in dfw | 15:41 |
| anteaya | yolanda: thank you | 15:41 |
| *** armax has quit IRC | 15:42 | |
| yolanda | anytime | 15:42 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_libvirt job https://review.openstack.org/269592 | 15:42 |
| anteaya | I think nodepool is the priority | 15:42 |
| mordred | yolanda: do you have bandwidth to see if the filesystems on the other servers are in good or bad state? | 15:42 |
| anteaya | <- cake | 15:42 |
| mtreinish | mordred: if you get a sec can you take a look at: https://review.openstack.org/234447 | 15:43 |
| yolanda | mordred, the ones you listed? | 15:43 |
| *** HeOS has quit IRC | 15:43 | |
| jeblair | yolanda, mordred, anteaya: o/ | 15:43 |
| mordred | mtreinish: WOW - devstack is emitting subunit? | 15:44 |
| jeblair | i get invalid issuer on https://status.rackspace.com/index/viewincidents?group=21&start=1453784400 | 15:44 |
| mordred | jeblair: yay! more help for yolanda that is not on an aeroplane | 15:44 |
| mtreinish | mordred: yep, I added that not too long ago | 15:44 |
| mordred | mtreinish: neat | 15:44 |
| *** jlanoux_ has joined #openstack-infra | 15:44 | |
| *** jlanoux has quit IRC | 15:44 | |
| mtreinish | I was thinking we might want d-g to emit subunit in the future too, but I'm not sure | 15:44 |
| jeblair | mordred, yolanda: cinder vl on afs01 looks ok | 15:47 |
| *** zeih has joined #openstack-infra | 15:47 | |
| yolanda | i'm checking elasticsearch ones, they look fine | 15:48 |
| jeblair | mordred, yolanda: mirror.dfw is fine | 15:49 |
| jeblair | nodepool is not file | 15:49 |
| jeblair | fine | 15:49 |
| yolanda | i remounted volume on graphite after the fsck | 15:49 |
| mordred | yolanda: ossum | 15:49 |
| jeblair | yolanda: have you started any repair work on nodepool? | 15:49 |
| yolanda | graphite is showing things now :) | 15:49 |
| yolanda | jeblair, no | 15:50 |
| *** placko_ has joined #openstack-infra | 15:50 | |
| jeblair | yolanda, mordred: i will start work on nodepool.o.o | 15:50 |
| yolanda | cool | 15:50 |
| *** gokrokve has joined #openstack-infra | 15:50 | |
| * mordred will watch yolanda and jeblair perform useful work | 15:50 | |
| sc68cal | mordred: if you can afford a context switch, could you take a look at https://review.openstack.org/#/c/267847/4/devstack-vm-gate.sh ? | 15:51 |
| *** dimtruck is now known as zz_dimtruck | 15:52 | |
| ansiwen | mordred, lifeless: I'm thinking of keeping the behaviour if the domain config is set. That is, if domain=foo is set, only that domain is compiled. If you want to enable automatic discovery, you have to leave it unset. what do you think? Aren't we on the safer side then? | 15:52 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh https://review.openstack.org/267117 | 15:53 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 15:53 |
| krotscheck | mordred: You can be useful over here ^^ :) | 15:53 |
| jeblair | okay | 15:53 |
| jeblair | i'm moving to #openstack-infra-incident | 15:53 |
| *** shardy_ has joined #openstack-infra | 15:53 | |
| *** Daisy has joined #openstack-infra | 15:53 | |
| *** shardy has quit IRC | 15:54 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update to oslo.config 3.4.0 https://review.openstack.org/272607 | 15:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh https://review.openstack.org/267117 | 15:56 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 15:56 |
| *** piet has quit IRC | 15:58 | |
| *** ihrachys has quit IRC | 15:59 | |
| *** jlanoux has joined #openstack-infra | 16:00 | |
| *** ihrachys has joined #openstack-infra | 16:00 | |
| *** maishsk has quit IRC | 16:00 | |
| adrienverge | mordred: Hi! About 271844 and 271845: the YAML linter is configurable, I added the option so it gives the same as `pyyaml.dump()`. It can even be made less strict. | 16:01 |
| adrienverge | Would you have a new fresh look? https://review.openstack.org/#/c/271844/ | 16:01 |
| adrienverge | I'm convinced there's a real value to check our yaml like our python. It would prevent both CI-breaking syntax errors and useless (but real) commits like b005206 Fix trailing whitespace, 3e78b33 Check indent etc. | 16:01 |
| *** jsavak has quit IRC | 16:01 | |
| *** jlanoux_ has quit IRC | 16:01 | |
| *** armax has joined #openstack-infra | 16:01 | |
| *** isaacb has quit IRC | 16:01 | |
| openstackgerrit | Tim Birkett proposed openstack-infra/jenkins-job-builder: Add RabbitMQ properties, trigger adn publisher https://review.openstack.org/272611 | 16:02 |
| *** placko_ has quit IRC | 16:04 | |
| amotoki | AJaeger: hi | 16:04 |
| *** ilyashakhat has quit IRC | 16:05 | |
| *** jsavak has joined #openstack-infra | 16:05 | |
| *** salv-orl_ has joined #openstack-infra | 16:06 | |
| *** ilyashakhat has joined #openstack-infra | 16:07 | |
| *** vgridnev has quit IRC | 16:08 | |
| *** mgould has joined #openstack-infra | 16:08 | |
| dhellmann | requirements cores, can I get a review on https://review.openstack.org/#/c/272607/ to update our oslo.config, please? | 16:08 |
| *** salv-orlando has quit IRC | 16:09 | |
| *** vgridnev has joined #openstack-infra | 16:09 | |
| *** burgerk has joined #openstack-infra | 16:10 | |
| *** r-daneel has joined #openstack-infra | 16:10 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Add fuel-plugin-etckeeper repository for Fuel https://review.openstack.org/260208 | 16:11 |
| *** jcoufal_ has joined #openstack-infra | 16:11 | |
| *** jcoufal has quit IRC | 16:12 | |
| *** amotoki has quit IRC | 16:16 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 16:16 |
| *** Daisy has quit IRC | 16:16 | |
| mgould | hi everyone! | 16:16 |
| prometheanfire | can I get a release cut for https://github.com/openstack-infra/glean/releases | 16:16 |
| prometheanfire | so that the simple-init element works on gentoo :P | 16:17 |
| *** baoli has quit IRC | 16:17 | |
| mgould | my patch is failing on check-osc-plugins because it's trying to build python-tuskarclient, which is now empty | 16:17 |
| mgould | the patch to check-osc-plugins to stop it doing that was merged at 1900UTC last night | 16:17 |
| mgould | is there any further step required to deploy that code to Jenkins workers, or do I just have to wait? | 16:18 |
| *** baoli has joined #openstack-infra | 16:18 | |
| *** Daisy has joined #openstack-infra | 16:18 | |
| *** david-lyle_ has quit IRC | 16:18 | |
| anteaya | dhellmann: I think you are looking at dims, lifeless and markmcclain | 16:19 |
| *** fawadkhaliq has quit IRC | 16:20 | |
| *** unicell has joined #openstack-infra | 16:21 | |
| prometheanfire | who handles the releases for glean? | 16:21 |
| anteaya | in gerrit, when you are signed in, people > list groups > use the search field | 16:22 |
| anteaya | <project>-release is the group that you are looking for | 16:22 |
| dims | dhellmann : +2 for now, please feel free to +A when jobs finish | 16:22 |
| dhellmann | dims : ty | 16:22 |
| *** tongli has joined #openstack-infra | 16:22 | |
| *** Daisy has quit IRC | 16:23 | |
| openstackgerrit | Merged openstack-infra/project-config: copy_puppet_logs: use SUDO to rename config/log files https://review.openstack.org/272518 | 16:23 |
| dhellmann | anteaya : thanks, I think some of the rest of infra have +2 there as well and I thought dims was unavailable today :-) | 16:23 |
| prometheanfire | anteaya: thanks | 16:23 |
| anteaya | dhellmann: ah yeah, infra-core is basically down to jeblair mordred and yolanda right now | 16:24 |
| *** dims has quit IRC | 16:24 | |
| anteaya | prometheanfire: welcome | 16:24 |
| dhellmann | anteaya : what did you do with the rest of them? ;-) | 16:24 |
| openstackgerrit | Merged openstack-infra/project-config: Set gate-designate-python34 voting https://review.openstack.org/272538 | 16:24 |
| jeblair | mordred is on a plane | 16:25 |
| anteaya | dhellmann: as other folks are doing other things as far as I know | 16:25 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 16:25 |
| jeblair | and i think yolanda is about to go to sleep | 16:25 |
| anteaya | jeblair: ah yes | 16:25 |
| openstackgerrit | Merged openstack-infra/project-config: Added new project for the OSA role break out https://review.openstack.org/272331 | 16:25 |
| *** aopincar has quit IRC | 16:25 | |
| anteaya | so jeblair is doing all the things | 16:25 |
| anteaya | thanks jeblair | 16:25 |
| dhellmann | jeblair , anteaya: that's much less alarming than I originally interpreted "down to" | 16:25 |
| yolanda | jeblair, not yet :) | 16:25 |
| yolanda | 5pm is not a good time to sleep | 16:25 |
| openstackgerrit | Merged openstack-infra/project-config: Move Manila LVM Tempest job to check queue https://review.openstack.org/272449 | 16:25 |
| anteaya | dhellmann: sorry, I chose words poorly | 16:26 |
| dhellmann | anteaya : I'm just teasing you | 16:26 |
| anteaya | dhellmann: :) | 16:26 |
| prometheanfire | ok, guess I need to wait for one of the infra-core people | 16:26 |
| anteaya | yolanda: feel like addressing the two folks with double gerrit accounts? | 16:26 |
| anteaya | yolanda: in backscroll? | 16:26 |
| anteaya | unless jeblair wants them | 16:26 |
| jeblair | there are some nodepool images that have been stuck in building for 2 days, i'm doing to delete their db records | 16:26 |
| *** jlanoux has quit IRC | 16:27 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 16:27 |
| anteaya | jeblair: ack | 16:27 |
| jeblair | greghaynes, clarkb: ^ | 16:27 |
| yolanda | anteaya, i am with some other non-infra things, but i can take a look in a while | 16:27 |
| anteaya | yolanda: thanks | 16:27 |
| anteaya | yolanda: thanks for your help with the file system issues | 16:27 |
| *** abregman is now known as abregman|brb | 16:28 | |
| *** jlanoux has joined #openstack-infra | 16:28 | |
| anteaya | nice work on static.o.o | 16:28 |
| yolanda | np... fun today :) | 16:28 |
| *** chlong is now known as chlong_zzz | 16:28 | |
| anteaya | :) | 16:28 |
| openstackgerrit | Merged openstack-infra/project-config: acl cleanup for openstack-chef project https://review.openstack.org/266843 | 16:28 |
| prometheanfire | yolanda: if you have time can you cut a release for glean? | 16:29 |
| yolanda | prometheanfire, i'd better wait for mordred or greghaynes for that | 16:29 |
| yolanda | i'm not much related with glean, apart from contributing | 16:29 |
| prometheanfire | sure | 16:29 |
| *** isaacb has joined #openstack-infra | 16:30 | |
| anteaya | jeblair: zuul memory usage is 10G of a 30G capacity, looking stable atm | 16:30 |
| jeblair | anteaya: thanks; if it climbs again, i will restart it without the connections patchs in order to collect more data | 16:30 |
| markmcclain | dhellmann: +2/A | 16:30 |
| dhellmann | markmcclain : thanks! | 16:31 |
| anteaya | jeblair: very good | 16:31 |
| anteaya | jeblair: jlk was asking about bluebox numbers if you have any space on your list | 16:31 |
| anteaya | jeblair: he was wondering if we see performance changes | 16:31 |
| jeblair | anteaya: we'll have to wait until the graphs at the bottom of http://grafana.openstack.org/dashboard/db/nodepool-blue-box populate again | 16:32 |
| * anteaya nods | 16:32 | |
| anteaya | jeblair: also wondering if you feel it is worthwhile to take a peek at java melody | 16:32 |
| anteaya | but not if you have something more important | 16:32 |
| jeblair | anteaya: i'm going to focus on nodepool | 16:33 |
| * anteaya nods | 16:33 | |
| anteaya | sorry to interrupt | 16:33 |
| jeblair | np | 16:33 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 16:33 |
| *** abregman|brb is now known as abregman | 16:34 | |
| *** abregman has quit IRC | 16:34 | |
| *** jflynn has joined #openstack-infra | 16:34 | |
| *** zhurong has joined #openstack-infra | 16:35 | |
| greghaynes | jeblair: someone else reported seeing fs corruption yesterday but I couldn't pin it down to being something unspecific to them | 16:35 |
| greghaynes | jeblair: so, thats scary | 16:35 |
| *** woodster_ has joined #openstack-infra | 16:36 | |
| greghaynes | they were seeing dib delete /dev nodes on the build host | 16:36 |
| jeblair | greghaynes: if it was in a job, those should be all local access, not iscsi | 16:36 |
| jeblair | but who knows :) | 16:36 |
| *** e0ne has joined #openstack-infra | 16:37 | |
| *** scheuran has quit IRC | 16:37 | |
| *** stevemar_znc is now known as stevemar | 16:38 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| greghaynes | jeblair: It was someone running nodepool as a third party ci | 16:39 |
| lucasagomes | hi all, sorry for the noob question... I've started a new project called ironic-staging-drivers, I got added to the gerrit ironic-staging-drivers-core group but I still have no -2...+2 powers | 16:39 |
| *** ilyashakhat has quit IRC | 16:39 | |
| lucasagomes | looked at the acl file in project-config it seems good | 16:39 |
| lucasagomes | there's anything I'm missing here? | 16:39 |
| greghaynes | jpr: ^ | 16:40 |
| greghaynes | jpr: You were seeing the /dev device deletion | 16:40 |
| openstackgerrit | Tim Birkett proposed openstack-infra/jenkins-job-builder: Remove unused variable in rabbitmq trigger https://review.openstack.org/272631 | 16:40 |
| prometheanfire | greghaynes: hi :D | 16:40 |
| greghaynes | prometheanfire: hello | 16:40 |
| anteaya | lucasagomes: you are in the core group, and the release team is also in the core group and you are in the release team | 16:41 |
| jeblair | greghaynes: there are 20 images that have not been uploaded to hpcloud more recently than 11 days ago | 16:41 |
| lucasagomes | anteaya, that's correct | 16:41 |
| jpr | greghaynes: yes. I think I just found something that could lead to different behavior of disk-image-create under nodepoold vs direct from the command line | 16:41 |
| lucasagomes | anteaya, https://review.openstack.org/#/admin/groups/1258,members | 16:41 |
| anteaya | I'm wondering if part of the issue is you are in both groups, so essentially listed twice in -core | 16:41 |
| prometheanfire | greghaynes: release glean please :D | 16:41 |
| lucasagomes | anteaya, https://review.openstack.org/#/admin/groups/1259,members | 16:41 |
| anteaya | lucasagomes: I haven't seen that before | 16:41 |
| prometheanfire | greghaynes: can't use it if it's not tagged I think | 16:42 |
| lucasagomes | anteaya, oh... I can try to remove the release team from the core | 16:42 |
| prometheanfire | the simple-init element | 16:42 |
| anteaya | or remove yourself the individual from -core | 16:42 |
| anteaya | leaveing yourself as the release team as a member of -core | 16:42 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 16:43 |
| *** zhurong has quit IRC | 16:43 | |
| jeblair | greghaynes: so i will upload all 20 of those manually | 16:43 |
| greghaynes | jeblair: :( I bet that is the same disconnext issue | 16:43 |
| *** zhurong has joined #openstack-infra | 16:43 | |
| greghaynes | jeblair: I am coding the disconnect handling stuff right now so hopefully itll be better soon | 16:43 |
| lucasagomes | anteaya, no dice... lemme try to log out and in again | 16:43 |
| greghaynes | prometheanfire: I don't have release perms on glean | 16:44 |
| lucasagomes | anteaya, :-/ no | 16:44 |
| *** diazjf has joined #openstack-infra | 16:44 | |
| greghaynes | prometheanfire: itll have to be mordred | 16:44 |
| jpr | greghaynes: the disk-image-create script sets up it's env based on the path to the script. nodepoold calls it with /usr/local/bin/disk-image-create which causes it to set using supporting scripts out of /usr/local/share/diskimage-builder . When I've called it manually under an ordinary user account I've not specified the path, so it defaults to my cwd | 16:44 |
| lucasagomes | anteaya, the ACL looks fine right? https://github.com/openstack-infra/project-config/blob/master/gerrit/acls/openstack/ironic-staging-drivers.config | 16:44 |
| prometheanfire | greghaynes: k, thanks :D | 16:44 |
| jpr | http://git.openstack.org/cgit/openstack/diskimage-builder/tree/bin/disk-image-create#n29 | 16:45 |
| *** apoorvad has joined #openstack-infra | 16:45 | |
| jeblair | greghaynes: ok. it will be good to at least log it, because we also can't actually function over the long run if we have disconnects | 16:45 |
| greghaynes | jpr: Yep, I really dont know how that would cause dib to break out of the dib chroot though | 16:45 |
| anteaya | lucasagomes: acl file looks like I would expect | 16:45 |
| jeblair | greghaynes: i mean, we should handle it if it happens, but then, after that, we need to make it stop happening, because we'll never get any images built | 16:45 |
| anteaya | lucasagomes: okay so only person with gerrit admin permisisons online right now is jeblair and he is kind of busy | 16:46 |
| jpr | greghaynes: haven't dug into the scripts in that dir but it seems to be the difference between success and failure, even for something as simple as the elements "ubuntu vm" | 16:46 |
| lucasagomes | anteaya, yeah I've no idea then :-/ | 16:46 |
| greghaynes | jeblair: yep. The beset theory on why it is happening is that we now blast all the image-upload jobs at once at gearman | 16:46 |
| anteaya | lucasagomes: stand by, grab a tea, perhaps work on something else for a bit | 16:46 |
| lucasagomes | anteaya, yeah doing that, no worries heh | 16:46 |
| greghaynes | jeblair: and that seems to just about coincide with when the disconnect occurs | 16:46 |
| anteaya | lucasagomes: neither do I, I think I need a gerrit admin to look and noone is availble who has time atm | 16:47 |
| lucasagomes | jeblair, when you have time (today or tomorrow) mind taking a look at this problem please? Thank you | 16:47 |
| anteaya | lucasagomes: thanks for your patience | 16:47 |
| *** nmagnezi_ has quit IRC | 16:47 | |
| dtroyer | quick LP/Gerrit question: a dev who used their employer's email for their account is changing jobs, and has a number of in-flight reviews. Is there something that should be done before giving up the email access when they change employers? (no, this is not me). | 16:47 |
| lucasagomes | anteaya, it's all good... thanks for the help | 16:47 |
| greghaynes | jpr: ok, so in theory if I just pip install diskimage-builder and then run from /usr/local/bin/disk-image-create I should see the same issue... | 16:47 |
| *** baoli has quit IRC | 16:47 | |
| anteaya | dtroyer: they need to add the email address to gerrit | 16:47 |
| anteaya | dtroyer: the non-employeer email | 16:47 |
| ansiwen | mordred: can I mock setup.cfg in the pbr tests? I just see examples that call setup.py as a normal command line tool. | 16:48 |
| jeblair | greghaynes: interesting. the number of jobs should not be at issue (other systems send far more), but it's good to have that as something suspicious for us to focus on | 16:48 |
| dtroyer | anteaya: to their existing account? | 16:48 |
| greghaynes | jeblair: so my suspicion there was it is something related to send/recv buffers filling on that socket, but my glancing over the gear code didnt reveal anything obvious | 16:48 |
| anteaya | dtroyer: yes | 16:48 |
| jpr | greghaynes: yes. | 16:48 |
| anteaya | to their existing gerrit account | 16:48 |
| *** mgould has quit IRC | 16:48 | |
| dtroyer | awesome. Thanks! | 16:48 |
| anteaya | dtroyer: keep the same launchpad creds | 16:48 |
| anteaya | dtroyer: don't add the new email to launchpad, just to gerrit | 16:48 |
| *** achanda has joined #openstack-infra | 16:49 | |
| jpr | greghaynes: it may not be breaking out as much as assuming to much.... the problematic command is http://git.openstack.org/cgit/openstack/diskimage-builder/tree/bin/disk-image-create#n282 | 16:49 |
| anteaya | dtroyer: they just need to remember their launchpad password and not need a reset going forward | 16:49 |
| *** sshnaidm has quit IRC | 16:49 | |
| greghaynes | jpr: ya, I wonder if we have a case where the image build fails, we clean up, then we do the mv anyways | 16:50 |
| jpr | greghaynes: ${TMP_BUILD_DIR}/built/* sweeps up the ./dev and ./proc bind mounted dirs in that build space and that's when the system /dev files disappear. | 16:50 |
| *** kzaitsev_mb has joined #openstack-infra | 16:50 | |
| *** dims has joined #openstack-infra | 16:50 | |
| greghaynes | jpr: ok, I need to refresh my memory on how that is supposed to work | 16:50 |
| anteaya | dtroyer: they also need to set their new email in gerrit to their prefered email address, as that is where summit passes and election emails get sent to, the gerrit prefered email | 16:50 |
| dtroyer | anteaya: ah, right. changing their LP account would be making a new one then… and I'm guessing changing Gerrit to match is also then making a new account? | 16:50 |
| jpr | greghaynes: it doesn't look like it's even gotten to a clean up phase. this looks more like the build work | 16:50 |
| anteaya | dtroyer: you can add as many emails to gerrit as you like, it doesn't create new gerrit accounts | 16:51 |
| greghaynes | jpr: we use trap exit to do some cleanup | 16:51 |
| greghaynes | jpr: so itll happen at any point | 16:51 |
| *** vgridnev has quit IRC | 16:51 | |
| anteaya | dtroyer: and yes, adding a new email to launchpad makes launchpad openid think it is a new account and creates a new gerrit account | 16:51 |
| *** adrienverge has quit IRC | 16:52 | |
| *** gokrokve has quit IRC | 16:52 | |
| anteaya | dtroyer: and the scripts for summit passes and election voting sends emails to the gerrit prefered email | 16:52 |
| dtroyer | right, thanks | 16:52 |
| anteaya | dtroyer: thanks for asking | 16:52 |
| *** baoli has joined #openstack-infra | 16:53 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 16:53 |
| jpr | greghaynes: ah ok. btw, i'm doing a command line build of the dpc image as well | 16:53 |
| *** pblaho has quit IRC | 16:53 | |
| greghaynes | jpr: what is dpc? | 16:53 |
| *** gokrokve has joined #openstack-infra | 16:54 | |
| jpr | greghaynes: sorry that's the default image name in the project-config-example nodepool.yaml which i'm using as part of openstackci | 16:54 |
| greghaynes | jpr: ah, ok | 16:54 |
| jpr | greghaynes: it builds with these elements "ubuntu vm openstack-repos puppet nodepool-base node-devstack" | 16:55 |
| greghaynes | You saw it with just 'ubuntu vm' though, correct? | 16:56 |
| openstackgerrit | Richard Theis proposed openstack/requirements: Bump openstacksdk to 0.7.4 https://review.openstack.org/272645 | 16:56 |
| *** dims has quit IRC | 16:56 | |
| anteaya | greghaynes: http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/nodepool/nodepool.yaml#n33 | 16:56 |
| *** fawadkhaliq has joined #openstack-infra | 16:57 | |
| *** mikelk has quit IRC | 16:57 | |
| greghaynes | anteaya: ty | 16:57 |
| anteaya | welcome | 16:57 |
| *** kushal has joined #openstack-infra | 16:58 | |
| *** hashar has quit IRC | 17:00 | |
| *** Swami has joined #openstack-infra | 17:01 | |
| *** jlanoux has quit IRC | 17:01 | |
| *** mriedem has quit IRC | 17:01 | |
| greghaynes | jpr: hrm, ubuntu vm with sytem installed diskimage-builder works fine for me still. | 17:01 |
| greghaynes | I wonder if this is a case where loopback devices get in a state which confuses dib | 17:01 |
| greghaynes | and dib does not handle the case well | 17:01 |
| *** Jeffrey4l has quit IRC | 17:02 | |
| *** yaume_ has joined #openstack-infra | 17:02 | |
| *** zeih_ has joined #openstack-infra | 17:02 | |
| *** jlanoux has joined #openstack-infra | 17:02 | |
| *** julim_ has joined #openstack-infra | 17:02 | |
| asselin | dpc is the short for devstack precise check...should be renamed.... | 17:02 |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-openstackci: Pass Java arguments to Jenkins master https://review.openstack.org/263021 | 17:03 |
| *** fawadk has joined #openstack-infra | 17:03 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 17:03 |
| *** mgould has joined #openstack-infra | 17:03 | |
| jpr | greghaynes: yeah. i'm wondering if this has something to do with the nodepool environment based of the openstackci work. I looked back at my notes and see that I tested an unadorned disk-image-create command from the nodepool account home dir. the same command from the ubuntu account home dir succeeds. "TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o /opt/nodepool_dib/6 ubuntu vm" | 17:03 |
| *** diazjf has quit IRC | 17:03 | |
| *** diazjf1 has joined #openstack-infra | 17:03 | |
| *** gokrokve has quit IRC | 17:03 | |
| *** markmcd1 has joined #openstack-infra | 17:04 | |
| anteaya | asselin: or keep the name and dream up a new meaning | 17:04 |
| *** kaisers1 has joined #openstack-infra | 17:04 | |
| *** hdd has joined #openstack-infra | 17:05 | |
| *** ashleighfarnham has joined #openstack-infra | 17:05 | |
| *** binbincong_ has joined #openstack-infra | 17:05 | |
| *** bpokorny has joined #openstack-infra | 17:06 | |
| *** zeih_ has quit IRC | 17:06 | |
| *** zeih_ has joined #openstack-infra | 17:06 | |
| *** isaacb has quit IRC | 17:07 | |
| *** kaisers has quit IRC | 17:07 | |
| *** zeih has quit IRC | 17:07 | |
| *** yaume has quit IRC | 17:07 | |
| *** stevemar has quit IRC | 17:07 | |
| *** fawadkhaliq has quit IRC | 17:07 | |
| *** peristeri has quit IRC | 17:07 | |
| *** julim has quit IRC | 17:07 | |
| *** markmcd has quit IRC | 17:07 | |
| *** nijaba has quit IRC | 17:07 | |
| *** binbincong has quit IRC | 17:07 | |
| *** jflynn has quit IRC | 17:07 | |
| *** jtomasek has quit IRC | 17:07 | |
| *** peristeri has joined #openstack-infra | 17:07 | |
| *** EricGonczer_ has joined #openstack-infra | 17:08 | |
| *** nijaba has joined #openstack-infra | 17:08 | |
| *** nijaba has joined #openstack-infra | 17:08 | |
| *** pblaho has joined #openstack-infra | 17:08 | |
| *** matrohon has quit IRC | 17:08 | |
| *** tphummel has joined #openstack-infra | 17:08 | |
| *** jtomasek has joined #openstack-infra | 17:08 | |
| *** stevemar_znc has joined #openstack-infra | 17:08 | |
| *** EricGonc_ has quit IRC | 17:08 | |
| anteaya | getting close to the end of our day here, doubt I will be present at infra meeting | 17:08 |
| *** jflynn has joined #openstack-infra | 17:09 | |
| *** signed8b_ has joined #openstack-infra | 17:10 | |
| *** jlanoux has quit IRC | 17:10 | |
| *** esikachev has quit IRC | 17:11 | |
| jklare | @yolanda hi, i just saw that you merged https://review.openstack.org/#/c/266843/ , do you know when the new groups mentioned in the patch will be created and usable? | 17:11 |
| *** ashleighfarnham has quit IRC | 17:12 | |
| *** jlanoux has joined #openstack-infra | 17:12 | |
| anteaya | jklare: so it sounds like this group needs to be disabled then: https://review.openstack.org/#/admin/groups/161,members | 17:13 |
| anteaya | jklare: the groups haven't been created yet: https://review.openstack.org/#/admin/groups/?filter=openstack-chef | 17:14 |
| jklare | @anteaya yeah, if its possible, i would add all active core reviewers to the newly created groups | 17:14 |
| jklare | @anteaya so if you add me first to the core and the release one, i should be able to take care of the rest | 17:15 |
| *** jordanP has quit IRC | 17:15 | |
| anteaya | well ideally we would just rename the group rather than create a new one and disable the old one | 17:15 |
| *** thorst_ has quit IRC | 17:15 | |
| *** EricGonc_ has joined #openstack-infra | 17:15 | |
| anteaya | however I don't have the ability to do that since I'm not a gerrit admin | 17:15 |
| jklare | anteaya sure, whatever is the best way to do that | 17:16 |
| *** jaypipes has quit IRC | 17:16 | |
| anteaya | and I'm feeling like our jeepyb manage-projects script will create the new group before anyone else can get to ti | 17:16 |
| jklare | works for me :) | 17:16 |
| jklare | as long as someone adds me to these groups and i can add the other cores so we can still merge stuff | 17:17 |
| anteaya | jklare: so check that link I offered and you can ask to be added to the new groups, have patience | 17:17 |
| *** openstackgerrit has quit IRC | 17:17 | |
| anteaya | someone can add you but it make take a while before someone has time to add you | 17:17 |
| *** openstackgerrit has joined #openstack-infra | 17:17 | |
| anteaya | patience is such a wonderful quality | 17:18 |
| anteaya | room is breaking for the evening, going offline, have a good day everyone | 17:18 |
| *** gokrokve has joined #openstack-infra | 17:18 | |
| *** jaime has joined #openstack-infra | 17:18 | |
| *** e0ne_ has joined #openstack-infra | 17:18 | |
| pleia2 | good morning | 17:19 |
| jklare | mmh... usually i am very patient, but in this case i intentionally commented on this patch, that the groups should be created first (or in the case of automatic creation soon after merging), since afterwards we will not be able to merge patches until its done :) | 17:19 |
| *** notnownikki has joined #openstack-infra | 17:20 | |
| *** sfinucan has quit IRC | 17:20 | |
| *** jflynn has quit IRC | 17:20 | |
| *** EricGonczer_ has quit IRC | 17:20 | |
| *** e0ne has quit IRC | 17:20 | |
| mordred | greghaynes: we need a new glean release? | 17:21 |
| *** maishsk has joined #openstack-infra | 17:21 | |
| greghaynes | mordred: yes, if we can get https://review.openstack.org/#/c/272253/ in first that would be awesome | 17:21 |
| mordred | greghaynes: looks great | 17:22 |
| yolanda | greghaynes ++ | 17:23 |
| *** piet has joined #openstack-infra | 17:23 | |
| yolanda | if we can also get https://review.openstack.org/267672 it will be even better | 17:23 |
| yolanda | mordred, SpamapS | 17:23 |
| *** [1]Thelo has joined #openstack-infra | 17:23 | |
| *** thorst_ has joined #openstack-infra | 17:23 | |
| openstackgerrit | Merged openstack-infra/glean: Fix bug in double configuring vlans on glean https://review.openstack.org/272253 | 17:23 |
| *** jsavak has quit IRC | 17:23 | |
| *** vgridnev has joined #openstack-infra | 17:24 | |
| mordred | yolanda: for that one, I feel like we had a localhost issue a while ago | 17:24 |
| yolanda | yay! | 17:24 |
| mordred | greghaynes: do you remember this? | 17:24 |
| jeblair | can we maybe only release glean if there are people around to fix it if it breaks? | 17:24 |
| mordred | jeblair: I agree | 17:24 |
| jeblair | because we're going on 2 weeks without image updates in some situations | 17:24 |
| greghaynes | I am around, I am currently swimming in fixing other breaks though | 17:25 |
| jeblair | and i'd like to get further ahead on that before we get behind | 17:25 |
| greghaynes | ya, I am totally fine with slowing things down while we catch up | 17:25 |
| SpamapS | +1 on releasing at the early part of human support windows. Sounds like we need better functional testing of glean too. | 17:25 |
| greghaynes | becuase I have a stack of about 3 bad breaks that need fixing ATM | 17:25 |
| *** Thelo has quit IRC | 17:25 | |
| *** [1]Thelo is now known as Thelo | 17:26 | |
| greghaynes | mordred: the localhost thing I dont remember... | 17:26 |
| greghaynes | yolanda: I left a comment on that second patch | 17:26 |
| yolanda | greghaynes, your comment makes a lot of sense, i'll fix it | 17:26 |
| jeblair | apropos of nothing, it seems to take a really long time to upload a centos7 image to hpcloud | 17:26 |
| yolanda | or we are going to get recursion problems again, this time with localhost | 17:26 |
| *** ashleighfarnham has joined #openstack-infra | 17:26 | |
| *** jsavak has joined #openstack-infra | 17:27 | |
| *** whoops has quit IRC | 17:27 | |
| *** jsavak has quit IRC | 17:27 | |
| *** whoops has joined #openstack-infra | 17:28 | |
| *** baoli has quit IRC | 17:28 | |
| *** thorst_ has quit IRC | 17:28 | |
| mordred | yolanda: something like that is my concern - as is the conern that we might fight with puppet on that one | 17:29 |
| jeblair | greghaynes: weird; the first image i uploaded to hpcloud took 10 mins; the second has been running for 40 minutes and is not done. they are the same size. | 17:29 |
| greghaynes | SpamapS: ohai | 17:29 |
| mordred | yolanda: but I do not have a specific issue in mind - just an uneasy feeling | 17:29 |
| greghaynes | SpamapS: jpr is the one who hit the dib issue first, and then infra hit it on the nodepool nodes | 17:29 |
| jeblair | mordred: see my comment to greghaynes ^ | 17:29 |
| *** baoli has joined #openstack-infra | 17:29 | |
| mordred | jeblair: I see this comment | 17:30 |
| *** stevemar_znc has quit IRC | 17:30 | |
| *** mika has quit IRC | 17:30 | |
| *** rvba has quit IRC | 17:30 | |
| jeblair | mordred, greghaynes: strace says data are still being moved | 17:30 |
| *** dmsimard|afk has quit IRC | 17:30 | |
| SpamapS | greghaynes: do we have artifacts? | 17:30 |
| *** EmilienM has quit IRC | 17:30 | |
| *** mika has joined #openstack-infra | 17:30 | |
| *** acruz has joined #openstack-infra | 17:30 | |
| greghaynes | SpamapS: http://paste.openstack.org/show/484948/ is a lot jpr posted last night | 17:31 |
| greghaynes | log | 17:31 |
| *** e0ne_ has quit IRC | 17:31 | |
| yolanda | mordred, puppet is working fine with that, not doubling entries for that. But we may found another problems. However i found that not having localhost entry created, prevented services like rabbitmq to run. We can add that on glean or if you think is risky, add the entry in our infra cloud in puppet | 17:31 |
| greghaynes | SpamapS: but not a lot more info to go on | 17:31 |
| SpamapS | greghaynes: AH, that's in the mv step | 17:32 |
| SpamapS | Haven't looked down that hole yet | 17:32 |
| *** maishsk has quit IRC | 17:32 | |
| greghaynes | yep | 17:32 |
| *** jlanoux has quit IRC | 17:32 | |
| greghaynes | jeblair: :/ | 17:32 |
| greghaynes | jeblair: throttling? | 17:32 |
| jeblair | greghaynes: you think hpcloud might do that? | 17:32 |
| greghaynes | jeblair: or rax | 17:32 |
| *** jsavak has joined #openstack-infra | 17:33 | |
| SpamapS | I think somebody finally told the cloud about its impending doom. | 17:33 |
| *** arxcruz has quit IRC | 17:33 | |
| jeblair | greghaynes: rax has bandwidth limits, but they are consistent | 17:33 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=982&rra_id=5&view_type=&graph_start=1453825022&graph_end=1453829274&graph_height=120&graph_width=500&title_font_size=10 | 17:33 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View https://review.openstack.org/272667 | 17:34 |
| *** dgonzalez has quit IRC | 17:34 | |
| jeblair | it looks like we got 100mbit for my first upload, and now we're stuck at 3mbit | 17:34 |
| mordred | jeblair: maybe threshold-induced throttling? | 17:34 |
| *** dmsimard has joined #openstack-infra | 17:34 | |
| jeblair | mordred: from whom? | 17:34 |
| greghaynes | jeblair: and it looks i/o bound? | 17:34 |
| mordred | jeblair: good question | 17:34 |
| *** stevemar_znc has joined #openstack-infra | 17:34 | |
| *** EmilienM has joined #openstack-infra | 17:34 | |
| *** dgonzalez has joined #openstack-infra | 17:34 | |
| *** rvba has joined #openstack-infra | 17:34 | |
| *** rvba has quit IRC | 17:34 | |
| *** rvba has joined #openstack-infra | 17:34 | |
| *** signed8b_ has quit IRC | 17:34 | |
| SpamapS | jeblair: expect HP Cloud to start stalling as sunset day gets nearer. ;) | 17:35 |
| mmedvede | yolanda: do you mind revisiting puppet-ciwatch review https://review.openstack.org/#/c/238606/? It is ready for approval | 17:35 |
| *** ayoung_ZZZzzzz is now known as ayoung | 17:35 | |
| jeblair | should we just shut down hpcloud? | 17:35 |
| * SpamapS imagines hpcloud asking over and over "Any word from the governor?" | 17:36 | |
| *** ayoung has quit IRC | 17:36 | |
| SpamapS | jeblair: well it's certainly not worth investing much time. | 17:36 |
| greghaynes | jeblair: I thought the plan was to do that any day now anyhow | 17:36 |
| mordred | jeblair: I'm tempted to agree with that idea - I don't think debugging it is valuable at this point | 17:36 |
| *** amaretskiy has quit IRC | 17:36 | |
| greghaynes | jeblair: I want to say the 28th was the plan? | 17:36 |
| greghaynes | which is two days | 17:36 |
| jeblair | i'm doing math to see if the image uploads would finish before then | 17:36 |
| *** jsavak has quit IRC | 17:37 | |
| *** tonytan4ever has quit IRC | 17:37 | |
| greghaynes | heh | 17:37 |
| SpamapS | Sunset is in 5 days. Though IIRC, it will still keep sort of working for a while beyond that. They're just not making uptime guarantees anymore... because.. before... they were. | 17:37 |
| jeblair | it will take 3 days to upload these images at 3mbit. | 17:38 |
| greghaynes | it is funny that if this is a WAN throttling issue we have the solution just in time to shut down the cloud needingit | 17:38 |
| SpamapS | greghaynes: Ok, so what I see in jpr's report is somehow dev got mounted in 'built', but unmount_image didn't catch it | 17:38 |
| mordred | jeblair: then I say we sunset early | 17:39 |
| greghaynes | SpamapS: aha | 17:39 |
| mordred | jeblair: if our original plan was to turn off in 2 days | 17:39 |
| zaro | morning | 17:39 |
| greghaynes | mordred: jeblair agreed on sunset now, even if it was 1 day it wouldn't be very useful | 17:39 |
| greghaynes | since we would get just a few hours of testing in then shutdown | 17:40 |
| *** achanda has quit IRC | 17:40 | |
| SpamapS | greghaynes: recent changes on 12/24 and 12/31 in unmount_dir | 17:40 |
| SpamapS | 6b7cf266 and 966f12dc | 17:40 |
| *** diana_clarke has joined #openstack-infra | 17:40 | |
| SpamapS | greghaynes: 966f12dc being your pipefail fix | 17:40 |
| SpamapS | greghaynes: Guessing that's it. | 17:41 |
| *** ayoung has joined #openstack-infra | 17:41 | |
| greghaynes | SpamapS: awesome :( | 17:41 |
| greghaynes | SpamapS: seems very plausible | 17:41 |
| openstackgerrit | Lukas Vacek proposed openstack/python-jenkins: Add optional kerberos support https://review.openstack.org/238535 | 17:41 |
| mordred | jeblair: hpcloud is the only provider providing devstack-centos7 fyi | 17:41 |
| jeblair | i guess no one ever got around to fixing that | 17:42 |
| prometheanfire | mordred: tag new glean? | 17:42 |
| greghaynes | prometheanfire: were fighting too many fires ATM to release that I think | 17:42 |
| mordred | prometheanfire: I'm in the air right now and not in a good positoin to debug things if it breaks things | 17:42 |
| *** vgridnev has quit IRC | 17:43 | |
| pleia2 | pabelanger: did you want to add the summit proposals to the agenda? (busy agenda, and it's time sensitive) | 17:43 |
| pabelanger | if we move default_data.json from stackalytics into project-config, do you think it will be feasible to preserve the commit history of that file? | 17:43 |
| pabelanger | pleia2: Oh, ya. Thanks. I should do that now | 17:43 |
| *** nmagnezi_ has joined #openstack-infra | 17:44 | |
| jeblair | pabelanger, ianw, greghaynes, fungi: anyone know the status of getting centos7 working in our other clouds? | 17:44 |
| SpamapS | greghaynes: I'm guessing && awk | grep on the conditional somehow failed | 17:44 |
| greghaynes | jeblair: I do not | 17:44 |
| greghaynes | jeblair: I didn't know there was an issue | 17:45 |
| pabelanger | jeblair: I believe ianw is focusing on fedora. I could see what is needed for centos7 dibs | 17:45 |
| prometheanfire | :( | 17:45 |
| *** cody-somerville has joined #openstack-infra | 17:45 | |
| prometheanfire | guess I'm stalled then | 17:45 |
| greghaynes | SpamapS: ok, if we want to back out the pipefail change for now I would be fine with it | 17:45 |
| *** jsavak has joined #openstack-infra | 17:45 | |
| greghaynes | SpamapS: since it should be totally safe to revert | 17:45 |
| *** unicell has quit IRC | 17:45 | |
| jeblair | greghaynes: i have this really vague idea that its dib images weren't working on rax or something | 17:46 |
| *** vgridnev has joined #openstack-infra | 17:46 | |
| prometheanfire | not doing frequent releases will make it harder to debug, last release was a while ago :| | 17:47 |
| greghaynes | prometheanfire: yep, things are very abnormably bad right now though | 17:47 |
| prometheanfire | yep | 17:47 |
| prometheanfire | been a bad last two weeks | 17:47 |
| reed | how does one access documentation of previous releases? links from http://docs.openstack.org/kilo/ all go to master | 17:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Turn off HPCloud https://review.openstack.org/272674 | 17:49 |
| *** sshnaidm has joined #openstack-infra | 17:49 | |
| mordred | jeblair: ^^ | 17:49 |
| *** jsavak has quit IRC | 17:50 | |
| jeblair | i'm doing some terribly bad things to the builder to clear out the bunch of upload jobs i enqueued | 17:50 |
| jeblair | mordred, greghaynes: i stopped and started the builder | 17:50 |
| jlk | Heh, would it be inappropriate for me to +1 the removal of HP Cloud? :D | 17:51 |
| jeblair | that aborted the slow job | 17:51 |
| greghaynes | SpamapS: Ya, the pipefail is extremely suspicious given that it there are a lot of cases related to loopback detachment with pipes and infra was seeing /dev/loop oddness | 17:51 |
| jeblair | jlk: certainly not if you also added 500 more instances to bluebox ;) | 17:51 |
| jlk | all it takes is blue dollars | 17:51 |
| nibalizer | oh is today the day | 17:51 |
| *** hockeynut is now known as hockeynut_otr | 17:51 | |
| mordred | nibalizer: not originally | 17:51 |
| jeblair | mordred, greghaynes: i restarted the builder, and the next upload is going much faster | 17:52 |
| greghaynes | jeblair: is it !hpcloud? | 17:52 |
| mordred | nibalizer: but we can't upload images to hp anymore without it taking actually multiple days | 17:52 |
| jeblair | greghaynes: all hpcloud | 17:52 |
| greghaynes | jeblair: :( | 17:52 |
| *** diazjf1 has quit IRC | 17:52 | |
| *** hockeynut_otr is now known as hockeynut | 17:52 | |
| mordred | nibalizer: o - maybe spoke too soon | 17:52 |
| *** tonytan4ever has joined #openstack-infra | 17:52 | |
| greghaynes | jeblair: and the builder wasn't chewing up cpu before or anything, correct? | 17:52 |
| jeblair | greghaynes: correct | 17:52 |
| openstackgerrit | Merged openstack/requirements: Cap PyKMIP version https://review.openstack.org/260599 | 17:52 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View https://review.openstack.org/272667 | 17:52 |
| *** electrofelix has quit IRC | 17:53 | |
| *** jsavak has joined #openstack-infra | 17:53 | |
| *** cody-somerville has quit IRC | 17:53 | |
| greghaynes | jeblair: I don't have any great ideas, then. All that code should be the same and is just blocking on openstack clients | 17:53 |
| *** dgonzalez has quit IRC | 17:53 | |
| jeblair | greghaynes: yeah, so maybe hpcloud had a sad and then got less sad | 17:53 |
| greghaynes | very possible | 17:53 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version https://review.openstack.org/229789 | 17:54 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client https://review.openstack.org/272479 | 17:54 |
| *** ifarkas has quit IRC | 17:54 | |
| greghaynes | we could have some smarts to timeout uploads... | 17:54 |
| *** zz_zz_ja is now known as zz_zz_zz_ja | 17:54 | |
| *** jaosorior has quit IRC | 17:55 | |
| greghaynes | jpr: Do you have the ability to reproduce the diskimage-builder failure locally? | 17:55 |
| greghaynes | jpr: if so I have a change i'd like you to test | 17:55 |
| *** dgonzalez has joined #openstack-infra | 17:55 | |
| jpr | greghaynes: yes. on a call back in a few | 17:55 |
| greghaynes | ok | 17:55 |
| SpamapS | I have to step away too | 17:55 |
| greghaynes | SpamapS: ok, ty for the debug help | 17:56 |
| SpamapS | jpr: try reverting 966f12dc4cd6f48f57a6d170d7fbccc5a28892a9 and running again, if you can. | 17:56 |
| SpamapS | I don't actually see the logic error | 17:56 |
| *** abregman has joined #openstack-infra | 17:56 | |
| SpamapS | but I think things may get a bit.. confusing.. with pipefails and the like. | 17:56 |
| SpamapS | we may want to make this very inelegant and brute-force... checking $? explicitly and maybe operating on temp files, just to be safe. | 17:57 |
| *** jsavak has quit IRC | 17:57 | |
| greghaynes | Yep, I think this is going to require a while of paging in how all our unmounting stuff works | 17:58 |
| *** abregman has quit IRC | 17:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 17:58 |
| *** abregman has joined #openstack-infra | 17:58 | |
| greghaynes | so hopefully someone can do that when we arent potentially harming people's build hosts | 17:58 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:59 | |
| *** hdd has quit IRC | 17:59 | |
| *** zhurong has quit IRC | 17:59 | |
| *** hockeynut_afk has joined #openstack-infra | 18:00 | |
| *** julim has joined #openstack-infra | 18:00 | |
| jeblair | greghaynes, mordred: on to the next upload; seems faster too | 18:00 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: PL released 1.8.0 of the apache module https://review.openstack.org/272676 | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| greghaynes | jeblair: awesome | 18:01 |
| nibalizer | pabelanger: ^^ | 18:01 |
| * greghaynes assumes hpcloud sad, then | 18:01 | |
| jeblair | greghaynes, mordred: so maybe we can leave hpcloud in for a bit and give folks some time to fix centos7? | 18:01 |
| greghaynes | jeblair: SGTM | 18:01 |
| jeblair | i'll keep babysitting it | 18:01 |
| *** flepied has quit IRC | 18:03 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:03 | |
| *** julim_ has quit IRC | 18:03 | |
| mordred | jeblair: ++ | 18:03 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:04 | |
| pabelanger | nibalizer: done | 18:04 |
| *** lucasagomes is now known as lucas-dinner | 18:04 | |
| *** hockeynut_afk has quit IRC | 18:05 | |
| openstackgerrit | Merged openstack/requirements: bump keystonemiddleware in upper constraints https://review.openstack.org/272183 | 18:06 |
| *** hockeynut_otr has joined #openstack-infra | 18:06 | |
| *** sridhar_ram has quit IRC | 18:06 | |
| *** abregman is now known as abregman|mtg | 18:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add centos7 nodes to OVH https://review.openstack.org/272678 | 18:08 |
| *** jsavak has joined #openstack-infra | 18:08 | |
| mordred | jeblair: ^^ there is just the addition of centos7 to ovh | 18:08 |
| *** yaume_ has quit IRC | 18:09 | |
| jeblair | mordred: you want to try both at once? | 18:10 |
| jeblair | mordred: why not try the dib first? | 18:10 |
| jeblair | though | 18:10 |
| mordred | jeblair: we have some jobs that use each type | 18:10 |
| *** yaume has joined #openstack-infra | 18:10 | |
| jeblair | now i wonder, why do we have a -dib and non-dib version of that? | 18:10 |
| mordred | right | 18:10 |
| *** achanda has joined #openstack-infra | 18:10 | |
| *** thorst_ has joined #openstack-infra | 18:11 | |
| *** yaume has quit IRC | 18:11 | |
| mordred | that's my problem - I do not know why we have both - so I figured that if we can build both in ovh, we can divorce the problem of dib/non-dib centos7 from the hp issue | 18:11 |
| jeblair | yeah, i guess that sounds okay | 18:11 |
| jeblair | i hope someone comes along at some point and does care | 18:11 |
| mordred | if we cannot build both in ovh, then we will obviously need to get centos7-dib solid | 18:11 |
| *** jistr has quit IRC | 18:11 | |
| *** zeih_ has quit IRC | 18:11 | |
| *** BobBall is now known as BobBall_AWOL | 18:11 | |
| mordred | (since I am confident in our ability to upload centos images to ovh) | 18:11 |
| mordred | ianw: ^^ when you are around | 18:12 |
| jeblair | i feel like using a snapshot image on ovh is retrograde | 18:12 |
| *** esikachev has joined #openstack-infra | 18:12 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: gnocchi: add openstack-server-release-jobs https://review.openstack.org/272680 | 18:12 |
| *** sriram has left #openstack-infra | 18:13 | |
| jeblair | mordred, greghaynes: i think we just switched back to slow mode on hpcloud | 18:14 |
| greghaynes | quick, hit the tubo button rest endpoint | 18:14 |
| greghaynes | turbo | 18:14 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use devstack subunit stream in output https://review.openstack.org/234447 | 18:14 |
| jeblair | greghaynes, mordred: basically, my process is i check strace and see if things scroll by fast or slow | 18:15 |
| mordred | jeblair: that's an excellent process | 18:15 |
| *** dguitarbite has quit IRC | 18:15 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Bump haproxy to 1.4.0 https://review.openstack.org/272681 | 18:15 |
| nibalizer | jeblair: heh | 18:16 |
| jeblair | mordred, greghaynes: restarted the builder, this upload is fast | 18:16 |
| *** dguitarbite has joined #openstack-infra | 18:16 | |
| greghaynes | jeblair: maybe we can remove all but centos7 from hpcloud? Seems like any time spent on babysitting hpcloud isn't well spent time | 18:17 |
| jeblair | greghaynes: mordred ^ that might be a good compromise | 18:17 |
| mordred | jeblair: kk | 18:17 |
| *** gokrokve has quit IRC | 18:17 | |
| *** unicell has joined #openstack-infra | 18:17 | |
| mordred | jeblair: add to the centos7-on-ovh patch? | 18:17 |
| jeblair | mordred: i aprvd it, so make a followup | 18:18 |
| mordred | kk | 18:18 |
| mordred | btw - what are we keeping precise around for? | 18:18 |
| jeblair | mordred, greghaynes: i'll adjust my process to only upload centos images to hpcloud | 18:18 |
| *** harlowja has joined #openstack-infra | 18:18 | |
| jeblair | no idea | 18:18 |
| jeblair | mordred: maybe kilo or something? | 18:18 |
| mordred | oh - infra puppet apply :) | 18:19 |
| jeblair | heh or that | 18:19 |
| jeblair | yeah, many of our servers are still precise | 18:19 |
| *** thorst_ has quit IRC | 18:19 | |
| mordred | I _think_ we're actually upgraded on almost all of them now | 18:19 |
| nibalizer | http://puppetboard.openstack.org/fact/lsbdistcodename mordred nope | 18:19 |
| mordred | nope. I'm wrong | 18:19 |
| nibalizer | we're more precise than aanything else | 18:19 |
| *** thorst_ has joined #openstack-infra | 18:19 | |
| mordred | well, let's not start that project yet | 18:20 |
| nibalizer | but some of this stuff probably is pretty easy | 18:20 |
| jeblair | it was handy when we had the recent security updates :) | 18:20 |
| nibalizer | apps.o.o and cacti.o.o should just go | 18:20 |
| jeblair | (which did not apply to precise) | 18:20 |
| *** stevemar_znc is now known as stevemar | 18:20 | |
| nibalizer | oh the whole elasticsearch cluster is on precise | 18:20 |
| nibalizer | jenkins masters and logstash workers | 18:21 |
| *** e0ne has joined #openstack-infra | 18:21 | |
| jeblair | i'm telnetting to geard and canceling upload jobs | 18:21 |
| jeblair | which i just remembered is a thing | 18:21 |
| *** fawadk has quit IRC | 18:24 | |
| *** thorst_ has quit IRC | 18:24 | |
| *** MarkAtwo_ has quit IRC | 18:25 | |
| *** mgould has quit IRC | 18:25 | |
| jpr | greghaynes: is that commit for disk-image-builder? | 18:25 |
| *** jsavak has quit IRC | 18:25 | |
| *** eil397 has joined #openstack-infra | 18:26 | |
| greghaynes | jpr: yes | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove everything from HP except centos7 https://review.openstack.org/272684 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove devstack-precise and devstack-precise-check https://review.openstack.org/272685 | 18:26 |
| jpr | greghaynes: btw, my manual build of dpc image worked fine from the ubuntu account. no messup of /dev | 18:26 |
| mordred | jeblair, nibalizer: ^^ | 18:26 |
| mordred | jeblair: turns out we do not use devstack-precise, only bare-precise | 18:27 |
| jeblair | nice! | 18:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add centos7 nodes to OVH https://review.openstack.org/272678 | 18:28 |
| jeblair | mordred: removal of devstack-precise-2-node snuck into the first patch. should we care enough to reroll? | 18:28 |
| *** hockeynut_otr has quit IRC | 18:28 | |
| mordred | jeblair: nah | 18:29 |
| *** achanda has quit IRC | 18:29 | |
| jeblair | k, noted and +2d | 18:30 |
| jeblair | nibalizer: ^? | 18:30 |
| mordred | jeblair: I do not see anything in jenkins jobs that use centos7-dib | 18:30 |
| mordred | jeblair: but I might be reading something wrong | 18:30 |
| *** angdraug has quit IRC | 18:30 | |
| mordred | jeblair: looked in jjb and zuul layuot | 18:30 |
| jeblair | mordred: i wonder if the problem is that centos7-dib wasn't working _at all_ ? | 18:30 |
| *** sabeen3 has quit IRC | 18:31 | |
| jeblair | i guess we're about to find out | 18:31 |
| mordred | :) | 18:31 |
| *** rossella_s has quit IRC | 18:31 | |
| jeblair | (this makes my babysitting centos7 dib uploads to hpcloud seem even more pointless) | 18:31 |
| nibalizer | looking | 18:31 |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| mordred | jeblair: double check me on the "we don't use them" - but if I'm right, I suggest we kill those too and just leave the centos7 snapshot images in hp | 18:32 |
| jpr | greghaynes: testing with dib 1.6.1.dev1 | 18:34 |
| jeblair | mordred: i agree, nothing uses devstack-centos7-dib | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove devstack-centos7-dib from HP https://review.openstack.org/272691 | 18:35 |
| mordred | jeblair: there you go then | 18:35 |
| *** jsavak has joined #openstack-infra | 18:35 | |
| jeblair | mordred: that will mean no dib images in hpcloud, so i can just drop what i'm doing entirely :) | 18:35 |
| mordred | jeblair: yup | 18:35 |
| *** zeih has joined #openstack-infra | 18:35 | |
| *** Swami has quit IRC | 18:36 | |
| *** jsavak has quit IRC | 18:36 | |
| nibalizer | mordred: we still have a couple devstack-precises in nodepool right now | 18:36 |
| *** jsavak has joined #openstack-infra | 18:36 | |
| nibalizer | ya we need to do the thing where we set min-ready to 0 | 18:37 |
| *** gokrokve has joined #openstack-infra | 18:37 | |
| nibalizer | then let it kill them off, then we can remove it entirely | 18:37 |
| *** flepied has joined #openstack-infra | 18:37 | |
| mordred | nibalizer: or maybe just delete them if you're looking at it atm | 18:37 |
| jeblair | let's merge the changes first, then update the config (which we have to do manually), then delete the nodes/images we don't need any more | 18:38 |
| jpr | greghaynes: foo. seemed like it was getting farther but failed in the end with the same mv command evacuating /dev | 18:38 |
| jeblair | nibalizer: stack at https://review.openstack.org/272691 is all yours if you want to review | 18:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Turn off HPCloud https://review.openstack.org/272674 | 18:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Cleanup after ourselves post HP sunset https://review.openstack.org/272692 | 18:39 |
| jeblair | nibalizer: can we skip the set to 0 first? | 18:39 |
| mordred | and there are those for the future | 18:39 |
| *** abregman|mtg is now known as abregman | 18:39 | |
| nibalizer | jeblair: so in gozer we would always set it to 0 with config changes | 18:40 |
| nibalizer | because if I just type delete right now it will make more | 18:40 |
| jeblair | nibalizer: yeah, i'm volunteering to fix things manually | 18:40 |
| nibalizer | oh okay | 18:40 |
| nibalizer | sure then | 18:40 |
| *** slogan621 has joined #openstack-infra | 18:40 | |
| jeblair | nibalizer: puppet is not running on that host anyway, so we will even need to make the config changes manually (which i'm also volunteering to do) | 18:41 |
| *** flepied1 has joined #openstack-infra | 18:41 | |
| nibalizer | ah | 18:41 |
| greghaynes | jeblair: for actually fixing nodepool->gear disconnects it might be worth trying colocating a separate geard on the nodepool host just for builders | 18:41 |
| *** jsavak has quit IRC | 18:41 | |
| *** gokrokve has quit IRC | 18:42 | |
| jeblair | greghaynes: we'll have to teach nodepool to connect to two different geards | 18:42 |
| greghaynes | jeblair: yep | 18:42 |
| *** jsavak has joined #openstack-infra | 18:42 | |
| greghaynes | jeblair: That should be pretty easy | 18:42 |
| jeblair | greghaynes: (i mean, for different purposes; it can already do so for ha purposes) | 18:42 |
| slogan621 | If I create an openstack project (e.g., openstack/foo), which is unofficial, is it required that contributors sign the individual contributor agreement, and/or the corporate contributor agreement? | 18:42 |
| *** ybathia has joined #openstack-infra | 18:42 | |
| jeblair | greghaynes: however, i'd like to reserve that option for later | 18:43 |
| *** ashleighfarnham has quit IRC | 18:43 | |
| greghaynes | jeblair: What alternative are you thinking? | 18:43 |
| jeblair | slogan621: see this change, which is not yet official, but some version of it is likely to be and should hopefully answer your question: https://review.openstack.org/269823 | 18:43 |
| *** sridhar_ram has joined #openstack-infra | 18:44 | |
| jeblair | greghaynes: well, i'd like to find the cause :) | 18:44 |
| *** flepied has quit IRC | 18:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove everything from HP except centos7 https://review.openstack.org/272684 | 18:44 |
| jeblair | greghaynes: let me rephrase that -- i would like the cause of the problem to be found :) | 18:44 |
| greghaynes | hah, yep, we probably should do that | 18:45 |
| *** zz_zz_zz_ja has quit IRC | 18:45 | |
| *** vgridnev has quit IRC | 18:45 | |
| jeblair | greghaynes: probably first steps are some more logging on the builder side (maybe gear debug logging); then if that doesn't tell us anything, we may need to turn on geard debug logging too. that must be done with care because it is a torrent | 18:45 |
| *** baoli has quit IRC | 18:45 | |
| greghaynes | jeblair: when we dug through logs geard had no idea about the disconnect until it tried to write to the connection to nodepool. We couldn't find the initial traceback on the nodepool side from when it disconnected | 18:46 |
| greghaynes | jeblair: I am not sure we have seen any disconnects from the builder | 18:46 |
| jeblair | oh | 18:46 |
| jeblair | nibalizer, mordred: i'm going to take a few mins to prepare for eating lunch at my keyboard; biab and will apply changes then | 18:47 |
| *** BobBall_AWOL has quit IRC | 18:48 | |
| *** achanda has joined #openstack-infra | 18:48 | |
| *** ashleighfarnham has joined #openstack-infra | 18:48 | |
| *** sdake has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove devstack-precise and devstack-precise-check https://review.openstack.org/272685 | 18:51 |
| *** trown is now known as trown|lunch | 18:51 | |
| *** ihrachys has quit IRC | 18:52 | |
| *** sdake has joined #openstack-infra | 18:53 | |
| *** angdraug has joined #openstack-infra | 18:53 | |
| *** pvaneck has joined #openstack-infra | 18:54 | |
| slogan621 | jeblair: thanks. If you would, look at http://docs.openstack.org/infra/manual/creators.html - the section Request Signing of ICLA says that I can specify in the project if contributors are required to sign the ICLA if the project is unofficial. It implies that no corporate license agreement is required, either. | 18:56 |
| jhesketh | Morning | 18:57 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:57 | |
| slogan621 | I assume that would effectively kill any chance of the project passing through governance, but my goal is to get code contributed, not necessarily be official. And I'm struggling with a company that has an issue t=with signing the corporate agreement. | 18:57 |
| *** jaosorior_ has joined #openstack-infra | 18:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove devstack-centos7-dib from HP https://review.openstack.org/272691 | 18:58 |
| *** diazjf has joined #openstack-infra | 18:58 | |
| *** krtaylor has quit IRC | 18:58 | |
| slogan621 | how would I go about getting a binding interpretation on this? What it says seems clear, contributors to unofficial projects can be allowed without signing anything. | 18:59 |
| *** flepied has joined #openstack-infra | 18:59 | |
| *** gokrokve has joined #openstack-infra | 18:59 | |
| pleia2 | meeting time :) | 19:00 |
| *** ybathia has quit IRC | 19:00 | |
| *** flepied1 has quit IRC | 19:01 | |
| *** jsavak has quit IRC | 19:02 | |
| *** jsavak has joined #openstack-infra | 19:03 | |
| *** krtaylor has joined #openstack-infra | 19:03 | |
| *** amitgandhinz has quit IRC | 19:04 | |
| *** shardy_ has quit IRC | 19:04 | |
| *** gokrokve has quit IRC | 19:04 | |
| *** amitgandhinz has joined #openstack-infra | 19:05 | |
| *** kzaitsev_mb has quit IRC | 19:06 | |
| *** dkranz has quit IRC | 19:08 | |
| *** erikwilson has joined #openstack-infra | 19:08 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/271902 | 19:08 |
| *** tmcpeak has joined #openstack-infra | 19:09 | |
| tmcpeak | hey all, it seems like that thing that pushes new versions to PyPI based on annotated tags might not have worked or is stuck, where's the right place to check? | 19:09 |
| *** hdd has joined #openstack-infra | 19:09 | |
| tmcpeak | for Bandit specifically | 19:10 |
| *** dkranz has joined #openstack-infra | 19:10 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 19:11 |
| greghaynes | jeblair: ^ that should help some, I still need to do the periodic cleanup for jobs which fall through the cracks | 19:11 |
| zaro | puppetboard.o.o seems broken, can any infra-root folks please take a look? | 19:11 |
| greghaynes | I also really want to test it a bit more | 19:11 |
| openstackgerrit | Sven Anderson proposed openstack-dev/pbr: Extend compile_catalog command to compile all .po files https://review.openstack.org/271562 | 19:12 |
| zaro | nv, looks like it just came back online. | 19:12 |
| nibalizer | zaro: it goes up and down | 19:12 |
| nibalizer | there is an item in the meeting agjenda to talk about it | 19:13 |
| pleia2 | it's on the agenda for today | 19:13 |
| zaro | is there a way to get older reports from puppetboard? | 19:13 |
| *** dims has joined #openstack-infra | 19:13 | |
| *** dkehn has quit IRC | 19:14 | |
| *** arahal_ has joined #openstack-infra | 19:15 | |
| ianw | greghaynes: i get the feeling gate-tempest-dsvm-ironic-pxe_ssh-dib is broken, but maybe only on ovh. who is responsible for this job? | 19:15 |
| greghaynes | ianw: it is, there is an ironic launchpad bug up for it | 19:16 |
| greghaynes | ianw: last I heard they were swimming in failures and trying to dig out | 19:16 |
| jeblair | slogan621: i would recommend writing to jbryce (jonathan bryce) at the openstack foundation with your concerns | 19:16 |
| *** Sukhdev has joined #openstack-infra | 19:17 | |
| ianw | greghaynes: oh, ok; can we kick it to non-voting while it's sorted out? | 19:17 |
| greghaynes | ianw: eh, I'd rather help them debug it if I had time... | 19:17 |
| *** zeih has quit IRC | 19:17 | |
| *** ybathia has joined #openstack-infra | 19:17 | |
| tmcpeak | infra wizards - can somebody please point me to the process that combs the openstack git repo for annotated commits and pushes to PyPI | 19:17 |
| *** dtardivel has quit IRC | 19:17 | |
| tmcpeak | or however that works, the things that does that? | 19:18 |
| tmcpeak | seems like it hasn't picked up my 0.17.1 Bandit release | 19:18 |
| greghaynes | ianw: looks like https://review.openstack.org/#/c/272564/ is a potential fix | 19:18 |
| greghaynes | ianw: https://bugs.launchpad.net/ironic/+bug/1537754 is the bug | 19:18 |
| openstack | Launchpad bug 1537754 in Ironic "Unable to associate floating IP 172.24.5.4 to fixed IP 10.1.0.5 for instance ..." [High,Triaged] | 19:18 |
| *** r-daneel has quit IRC | 19:18 | |
| *** ybathia has quit IRC | 19:19 | |
| *** ybathia has joined #openstack-infra | 19:19 | |
| *** arahal_ has quit IRC | 19:19 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: Unified Mirrors https://review.openstack.org/252678 | 19:19 |
| slogan621 | jeblair: thanks | 19:20 |
| ianw | greghaynes: well that actually appears to be a different problem to http://logs.openstack.org/62/271862/5/check/gate-tempest-dsvm-ironic-pxe_ssh-dib/340baed/ ... | 19:20 |
| *** dkehn has joined #openstack-infra | 19:20 | |
| greghaynes | ianw: oh, a different fail for that job | 19:20 |
| *** dprince has quit IRC | 19:21 | |
| greghaynes | ianw: https://etherpad.openstack.org/p/IronicWhiteBoard is useful for finding possible bug references | 19:21 |
| *** achanda has quit IRC | 19:21 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: Unified Mirrors https://review.openstack.org/252678 | 19:22 |
| AJaeger | ianw: ironic team is and they're working on it... | 19:22 |
| AJaeger | ianw: all of ironic is blocked | 19:22 |
| *** sabeen1 has joined #openstack-infra | 19:22 | |
| *** achanda has joined #openstack-infra | 19:23 | |
| *** dims has quit IRC | 19:24 | |
| ianw | AJaeger / greghayes : ok, sounds like SNAFU | 19:24 |
| greghaynes | ianw: yep. I think they could definitely use some help | 19:25 |
| ianw | SpamapS / greghaynes : if I could get some eyes on https://review.openstack.org/#/c/271862/ modulo the ironic issues, that would be good. required for devstack to work on F23 | 19:25 |
| *** Swami has joined #openstack-infra | 19:25 | |
| *** achanda has quit IRC | 19:25 | |
| greghaynes | ianw: ok | 19:25 |
| AJaeger | ianw, greghaynes https://review.openstack.org/270356 could use some review love as well, please - but is blocked like others... | 19:26 |
| greghaynes | AJaeger: ah, that | 19:27 |
| jroll | ianw: greghaynes: we have a revert of a neutron commit that seems suspect, was waiting for jenkins | 19:27 |
| * jroll checks up on it | 19:27 | |
| ianw | jroll: latest comment in there seems to say it's not that ... | 19:28 |
| greghaynes | jroll: I saw neutron dev say they think it is suspect | 19:28 |
| jroll | I'm just saying what I know right now. | 19:28 |
| jroll | I see the comments | 19:28 |
| *** diana_clarke has quit IRC | 19:28 | |
| *** simondodsley has quit IRC | 19:28 | |
| greghaynes | jroll: hah, ok :) | 19:28 |
| jroll | and I guess I sit and wait for ryan to post a different patch | 19:28 |
| docaedo | Any chance someone would like to take a quick look at an update for the app-catalog periodic job? Not urgent, but maybe some infra folk would like a quick break from doing all the hard thinking they've been up to lately :) https://review.openstack.org/#/c/269935/ | 19:29 |
| *** sabeen3 has joined #openstack-infra | 19:29 | |
| *** thorst_ has joined #openstack-infra | 19:29 | |
| jroll | ianw: the last logs you posted is another bug I'm working on but is more intermittent | 19:30 |
| AJaeger | thanks, ianw and greghaynes ! | 19:30 |
| *** bpokorny_ has joined #openstack-infra | 19:30 | |
| jroll | greghaynes: ianw: I guess I'm not opposed to making our job nv on dib though, I don't want to block y'all | 19:31 |
| *** bpokorny_ has quit IRC | 19:31 | |
| greghaynes | jroll: eh, that job has been really stable, it only seems to fail for actual breaks (like this) | 19:31 |
| greghaynes | so I like it | 19:31 |
| *** sabeen1 has quit IRC | 19:31 | |
| *** bpokorny_ has joined #openstack-infra | 19:31 | |
| *** mkoderer_cloud has quit IRC | 19:31 | |
| jroll | greghaynes: okay, our jobs aren't super stable lately is why I mention it | 19:32 |
| greghaynes | jroll: well, relative to dib tests... | 19:32 |
| jroll | hah | 19:32 |
| jroll | fair enough :) | 19:32 |
| *** thorst_ has quit IRC | 19:33 | |
| *** thorst_ has joined #openstack-infra | 19:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Don't use internap for 2-node labels" https://review.openstack.org/269763 | 19:34 |
| *** bpokorny has quit IRC | 19:34 | |
| AJaeger | jeblair: I've rebased your change, is it safe to merge this now ? ^ | 19:34 |
| tmcpeak | hey all, can somebody please help me figure out why my annotated tag didn't get pushed to PyPI | 19:34 |
| jeblair | AJaeger: afaik :) | 19:35 |
| AJaeger | tmcpeak: which repo? Which tag? | 19:35 |
| AJaeger | jeblair: then let's merge ;) | 19:35 |
| *** rlandy has quit IRC | 19:35 | |
| tmcpeak | AJaeger: thank you, https://github.com/openstack/bandit/releases/tag/0.17.1 | 19:35 |
| AJaeger | tmcpeak: nit git.openstack.org is our master ;) | 19:36 |
| jeblair | AJaeger: ++ | 19:36 |
| tmcpeak | AJaeger: ahh, I learn something new e'ryday :) | 19:36 |
| tmcpeak | http://git.openstack.org/cgit/openstack/bandit/tag/?h=0.17.1 | 19:36 |
| *** thorst_ has quit IRC | 19:38 | |
| AJaeger | strange, 0.17.1 is not in my local repo ;( | 19:39 |
| tmcpeak | AJaeger: huh? | 19:39 |
| AJaeger | not sure what's broken there... | 19:39 |
| tmcpeak | where's your local repo? | 19:39 |
| *** daemontool_ has joined #openstack-infra | 19:40 | |
| greghaynes | jpr: Any updates on that revert? | 19:40 |
| greghaynes | jpr: oh, I missed, you said it still failed | 19:40 |
| jpr | greghaynes: yeah. it didn't work (noted above) :/ | 19:40 |
| greghaynes | jpr: can you paste some logs? | 19:40 |
| jpr | yes | 19:40 |
| *** rlandy has joined #openstack-infra | 19:41 | |
| greghaynes | jpr: and also that reverted just adding set -o pipefail, correct? | 19:41 |
| jpr | in a meeting. i'll get the log out in a few. | 19:41 |
| greghaynes | jpr: ok, ty | 19:41 |
| *** dprince has joined #openstack-infra | 19:42 | |
| AJaeger | yolanda: your move of the log files resulted in an extra logs directory, see http://logs.openstack.org/logs/ and http://logs.openstack.org/ | 19:42 |
| *** daemontool has quit IRC | 19:43 | |
| *** eil397 has quit IRC | 19:43 | |
| *** adrienverge has joined #openstack-infra | 19:43 | |
| *** ociuhandu has quit IRC | 19:44 | |
| AJaeger | anybody that can help me to find the logs of a release job, please? | 19:45 |
| *** ashleighfarnham has quit IRC | 19:46 | |
| *** eil397 has joined #openstack-infra | 19:46 | |
| *** salv-orl_ has quit IRC | 19:46 | |
| AJaeger | infra-root, I need some help - bandit has according to http://git.openstack.org/cgit/openstack/bandit a 0.17.1 tag - but I do not see that in my local tree. My origin is https://git.openstack.org/openstack/bandit | 19:46 |
| AJaeger | this is on two machines ;( I do not see the 0.17.1 tag at all. Do we have a corruption on one of our gits? | 19:47 |
| * AJaeger checks out again from scratch... | 19:47 | |
| *** dizquierdo has quit IRC | 19:47 | |
| tmcpeak | AJaeger: thanks for looking into it | 19:48 |
| AJaeger | fresh checkout is fine ;( | 19:48 |
| zaro | nibalizer: is there a way to get older than last ten puppet reports ? | 19:48 |
| *** esikachev has quit IRC | 19:50 | |
| *** HeOS has joined #openstack-infra | 19:50 | |
| nibalizer | um i think so | 19:50 |
| nibalizer | in puppetboard do you mean? | 19:50 |
| *** gokrokve has joined #openstack-infra | 19:52 | |
| *** edmondsw has quit IRC | 19:53 | |
| nibalizer | ya it looks like our version doesn't support the config option but more recent versions do | 19:53 |
| *** BobBall_AWOL has joined #openstack-infra | 19:55 | |
| *** gokrokve has quit IRC | 19:56 | |
| *** trown|lunch is now known as trown | 19:56 | |
| AJaeger | https://jenkins.openstack.org/view/All/job/bandit-pypi-both-upload/ does not show the job at all, last run was for 0.17.0. I need some help on what's going on here... | 19:57 |
| nibalizer | also related to puppetdb crashing http://paste.openstack.org/show/485071/ | 19:58 |
| *** rcernin has quit IRC | 19:58 | |
| *** rfolco has quit IRC | 19:59 | |
| *** aysyd has quit IRC | 19:59 | |
| pleia2 | nobody died while I was chairing \o/ | 20:00 |
| *** rockyg has joined #openstack-infra | 20:00 | |
| pleia2 | (then again mordred's plane was still landing, so maybe the jury is out on that) | 20:01 |
| AJaeger | pleia2: just bandit 0.17.1 release confuses me. | 20:01 |
| *** EricGonc_ has quit IRC | 20:01 | |
| AJaeger | My existing repos do not show the change nor the tag when pulling, a new checkout looks fine. | 20:01 |
| AJaeger | But worse: No release to pypi done - see jenkins link above | 20:01 |
| pleia2 | elizabeth@r2d2$:~/bandit(master)$ git tag | grep 0.17.1 | 20:02 |
| pleia2 | 0.17.1 | 20:02 |
| *** gokrokve has joined #openstack-infra | 20:02 | |
| pleia2 | ^^ fresh clone | 20:02 |
| zaro | pleia2: nice job today | 20:02 |
| AJaeger | anybody with an old clone? | 20:02 |
| pleia2 | zaro: thank you | 20:02 |
| *** jsavak has quit IRC | 20:02 | |
| *** ashleighfarnham has joined #openstack-infra | 20:02 | |
| *** EricGonczer_ has joined #openstack-infra | 20:03 | |
| pleia2 | I'm off to lunch now | 20:03 |
| *** jsavak has joined #openstack-infra | 20:03 | |
| *** matrohon has joined #openstack-infra | 20:03 | |
| zaro | nibalizer: so how can review puppet reports for review.o.o around 1/15/2016 ? | 20:03 |
| *** austin81 has joined #openstack-infra | 20:03 | |
| AJaeger | tmcpeak: I can't find the logs, so this needs infra-root help. Looks like the job never run and I don't know why.. | 20:06 |
| tmcpeak | AJaeger: ok cool, anything I should do to get the ball rolling on that? | 20:06 |
| AJaeger | tmcpeak: ask here - I'll leave soon... | 20:06 |
| *** vgridnev has joined #openstack-infra | 20:06 | |
| pabelanger | jeblair: so, centos-7 dib worked as expected locally. will upload to clouds to see what happens | 20:07 |
| tmcpeak | AJaeger: ok will do, thanks for taking a look | 20:07 |
| *** kzaitsev_ws has quit IRC | 20:07 | |
| *** kzaitsev_ws has joined #openstack-infra | 20:07 | |
| *** ashleighfarnham has quit IRC | 20:10 | |
| *** matrohon has quit IRC | 20:10 | |
| *** sdake has quit IRC | 20:10 | |
| rbrndt_ | Having trouble in a gate job, horizon install fails with "+ django-admin.py compress --force | 20:11 |
| rbrndt_ | 2016-01-26 17:18:47.906 | /opt/stack/new/devstack/lib/horizon: line 153: django-admin.py: command not found" | 20:11 |
| *** sridhar_ram has quit IRC | 20:11 | |
| *** diazjf has quit IRC | 20:11 | |
| rbrndt_ | Not sure how to solve this or how its related to my changes in https://review.openstack.org/#/c/269805/6 | 20:11 |
| prometheanfire | mordred: why do source and repo install seem switched up to me? https://github.com/openstack/diskimage-builder/tree/master/elements/simple-init/install.d | 20:11 |
| prometheanfire | repo install uses 'pip install /tmp/glean.git' source install uses 'pip install glean' | 20:12 |
| zaro | yolanda: ping | 20:13 |
| *** pradk has quit IRC | 20:14 | |
| *** pradk_ has joined #openstack-infra | 20:14 | |
| *** matrohon has joined #openstack-infra | 20:14 | |
| *** ldnunes_ has quit IRC | 20:14 | |
| *** pradk_ is now known as pradk | 20:14 | |
| openstackgerrit | greghaynes proposed openstack-infra/gear: Guard against double shutdown in client https://review.openstack.org/272717 | 20:15 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 20:15 |
| *** krtaylor has quit IRC | 20:16 | |
| nibalizer | zaro: im not sure we have those | 20:17 |
| nibalizer | why do you need them? | 20:17 |
| *** rguillebert has quit IRC | 20:17 | |
| nibalizer | if its super critical me or another rooter can make api requests against puppetdb directly but its kindof a lot of work | 20:17 |
| *** notnownikki has quit IRC | 20:18 | |
| *** EricGonc_ has joined #openstack-infra | 20:18 | |
| *** adrienverge has quit IRC | 20:18 | |
| nibalizer | Hunner: i think our custom 'submit to puppetdb' python script is causing this http://paste.openstack.org/show/485071/ maybe? thats puppetdb.log | 20:18 |
| *** ashleighfarnham has joined #openstack-infra | 20:19 | |
| *** flwang1 has joined #openstack-infra | 20:20 | |
| *** smarcet has quit IRC | 20:21 | |
| *** EricGonczer_ has quit IRC | 20:21 | |
| *** gildub has joined #openstack-infra | 20:22 | |
| AJaeger | pabelanger: I've updated the talk description, please tell me whether you like it or whehter it needs more work. | 20:22 |
| *** rguillebert has joined #openstack-infra | 20:22 | |
| AJaeger | pabelanger: thanks for pulling this together | 20:22 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 20:23 |
| *** ashleighfarnham has quit IRC | 20:23 | |
| *** ashleighfarnham has joined #openstack-infra | 20:24 | |
| Hunner | nibalizer: Well, it sounds like the report it's trying to submit already exists? | 20:24 |
| Hunner | I don't think the python script can help that | 20:25 |
| *** dane-fichter_ has quit IRC | 20:26 | |
| *** nikhil has quit IRC | 20:27 | |
| *** nikhil has joined #openstack-infra | 20:28 | |
| *** BobBall_AWOL has quit IRC | 20:29 | |
| *** diazjf has joined #openstack-infra | 20:31 | |
| *** derekh has joined #openstack-infra | 20:31 | |
| *** rcernin has joined #openstack-infra | 20:33 | |
| *** flwang has left #openstack-infra | 20:33 | |
| tmcpeak | hey all, I'm trying to figure out why Bandit 0.17.1 hasn't automatically been pushed to PyPI by OpenStack CI despite my release of an annotated tag. AJaeger took a look earlier and seems to think something might be wrong with the jobs doing it. Any infra-root available to take a look please? | 20:33 |
| *** jsavak has quit IRC | 20:33 | |
| *** notnownikki has joined #openstack-infra | 20:33 | |
| *** jsavak has joined #openstack-infra | 20:34 | |
| *** gokrokve has quit IRC | 20:34 | |
| jpr | greghaynes: uploaded image build log from the failed build with 1.6.1.dev1 | 20:36 |
| *** harlowja_ has joined #openstack-infra | 20:36 | |
| *** gokrokve has joined #openstack-infra | 20:36 | |
| *** harlowja has quit IRC | 20:37 | |
| *** yamahata has quit IRC | 20:37 | |
| *** hashar has joined #openstack-infra | 20:37 | |
| *** signed8b_ has joined #openstack-infra | 20:38 | |
| clarkb | tmcpeak: http://logs.openstack.org/f4/f4961732992df5b9560b268114ddf8b1b53501f7/release/bandit-tarball/beb1b8c/console.html#_2016-01-26_18_55_44_313 | 20:40 |
| tmcpeak | clarkb: huh, any idea why that would be? http://git.openstack.org/cgit/openstack/bandit/tag/?h=0.17.1 | 20:41 |
| clarkb | not sure why that happened and I am in a van in anothwr country unable to debug further | 20:41 |
| *** sridhar_ram has joined #openstack-infra | 20:41 | |
| tmcpeak | clarkb: good times :) | 20:42 |
| AJaeger | clarkb: thanks for that link. So, that confirms my problem: udpating a local checkout does not get me 0.17.1, only a fresh checkout does. | 20:42 |
| AJaeger | is the repo somehow broken? | 20:42 |
| clarkb | so things to check | 20:43 |
| clarkb | is there a branch with that name? | 20:43 |
| *** sdake has joined #openstack-infra | 20:44 | |
| tmcpeak | there's an annotated tag with that name yes | 20:44 |
| AJaeger | no branch for git branch --remote | 20:44 |
| *** harlowja has joined #openstack-infra | 20:45 | |
| clarkb | I dont think you can have ambiguous overlapping names | 20:45 |
| *** harlowja_ has quit IRC | 20:45 | |
| tmcpeak | clarkb: what do you mean? | 20:45 |
| openstackgerrit | Tim Burke proposed openstack-infra/meetbot: Make timestamps link to their own anchors https://review.openstack.org/272726 | 20:46 |
| tmcpeak | this is the same tag naming format we've been using for a while now | 20:46 |
| clarkb | I mean you can only have one thing called 0.17.1 | 20:46 |
| tmcpeak | I believe that's what we have | 20:46 |
| openstackgerrit | Tim Burke proposed openstack-infra/meetbot: Add __str__ methods for items https://review.openstack.org/272727 | 20:46 |
| *** esikachev has joined #openstack-infra | 20:46 | |
| openstackgerrit | Richard Theis proposed openstack/requirements: Bump openstacksdk to 0.7.4 https://review.openstack.org/272645 | 20:46 |
| prometheanfire | mordred: is is possible to even run the simple-init datasource from git? it states it's missing the git repo (didn't seem to fetch) | 20:46 |
| *** salv-orlando has joined #openstack-infra | 20:47 | |
| tmcpeak | there is only one "0.17.1" tag if that's what you mean clarkb | 20:47 |
| *** sdake has quit IRC | 20:47 | |
| clarkb | I mean only one ref at all | 20:47 |
| clarkb | branch tag etc | 20:48 |
| pabelanger | ianw: regarding 272590 I would be also okay with sane defaults for gearman, like localhost or something | 20:48 |
| prometheanfire | mordred: you can test with prepending your dib call with DIB_DEFAULT_INSTALLTYPE=repo | 20:48 |
| tmcpeak | yeah, we only deal in the 'master' branch in Bandit, that should be fine | 20:48 |
| pabelanger | AJaeger: roger | 20:48 |
| *** sdake has joined #openstack-infra | 20:49 | |
| AJaeger | pabelanger: feel free to edit... | 20:49 |
| pabelanger | AJaeger: looks good to me | 20:49 |
| AJaeger | thanks | 20:49 |
| prometheanfire | mordred: ah, because the source-repo element checks for INSTALL_TYPE being source, so that switch is breaking things | 20:50 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Remove gate-openstack-ansible-lint https://review.openstack.org/272731 | 20:50 |
| *** signed8__ has joined #openstack-infra | 20:50 | |
| *** esikachev has quit IRC | 20:51 | |
| tmcpeak | ok so it seems clear that it wasn't able to grab that tagged version, which AJaeger also found locally | 20:51 |
| *** zul has quit IRC | 20:52 | |
| tmcpeak | the question is why | 20:52 |
| AJaeger | the strange thing is that the top of my repo is: Merge "This fixes the blacklist to look like a plugin so we can filter it" | 20:53 |
| *** signed8b_ has quit IRC | 20:53 | |
| *** pradk has quit IRC | 20:53 | |
| *** derekh has quit IRC | 20:53 | |
| *** krtaylor has joined #openstack-infra | 20:54 | |
| *** pradk has joined #openstack-infra | 20:54 | |
| *** matrohon has quit IRC | 20:54 | |
| tmcpeak | AJaeger: yeah so I did something a little unconventional | 20:54 |
| AJaeger | tmcpeak: when I run git show on 0.17.1, it shows me a commmit: | 20:54 |
| AJaeger | c364408faa25ccdc378b72c0a9e2150d20a94cba | 20:54 |
| AJaeger | and when I run git show on that one I get | 20:54 |
| AJaeger | fatal: bad object c364408faa25ccdc378b72c0a9e2150d20a94cba | 20:54 |
| *** diazjf has quit IRC | 20:54 | |
| AJaeger | but that's on my existing repo - it works on the fresh clone | 20:55 |
| *** bknudson has joined #openstack-infra | 20:55 | |
| *** nmagnezi_ has quit IRC | 20:55 | |
| tmcpeak | AJaeger: I took 0.17.0 that was released in December | 20:55 |
| AJaeger | give me a few minuts to check something... | 20:55 |
| tmcpeak | cherry picked the commit for one new feature, and released 0.17.1 from that | 20:55 |
| clarkb | did you push a commit that is not in gerrit? | 20:56 |
| tmcpeak | so there are a bunch of things that happened since December that 0.17.1 doesn't have, but it does have a fairly recent commit | 20:56 |
| clarkb | er push a tag for a commit | 20:56 |
| clarkb | ? | 20:57 |
| AJaeger | tmcpeak: git log 0.17.0..HEAD |grep c364408faa25ccdc378b72c0a9e2150d20a94cba | 20:57 |
| *** david_cu has joined #openstack-infra | 20:57 | |
| AJaeger | That finds nothing for me. | 20:57 |
| AJaeger | But c36... is the one you tagged | 20:57 |
| AJaeger | And that's on the fresh checkout. So, you tagged something that is not in master it seems | 20:57 |
| tmcpeak | clarkb: no, the commit that I cherry-picked is in Gerrit | 20:57 |
| AJaeger | on the other hand, https://review.openstack.org/#/c/271465/ merged | 20:58 |
| clarkb | yes but the resulting commit isnt the same commit | 20:58 |
| clarkb | cherry picks alwayd make a new commit | 20:58 |
| *** matrohon has joined #openstack-infra | 20:58 | |
| AJaeger | tmcpeak: check 271465, it has a different commit ID then what you tagged | 20:58 |
| clarkb | so uf you did not push that commit to gerrit and get it merged that may be the issue | 20:58 |
| tmcpeak | clarkb: ahh | 20:58 |
| AJaeger | tmcpeak: start with a clean repo again: git reset --hard origin/master | 20:59 |
| AJaeger | and tag 0.17.2 now, don't reuse 0.17.1... | 20:59 |
| *** hdd has quit IRC | 20:59 | |
| tmcpeak | ok, then we'll have to create a branch I suppose | 20:59 |
| AJaeger | tmcpeak: ah, you want to apply a single patch on top of 0.17.0? ;( | 20:59 |
| tmcpeak | AJaeger: ok, can I just delete 0.17.1? | 20:59 |
| AJaeger | use a branch... | 20:59 |
| AJaeger | tmcpeak: you cannot | 20:59 |
| tmcpeak | AJaeger: ok, cool, I'll give that a shot | 21:00 |
| AJaeger | tmcpeak: write up first what you do and have it sanity-checked by clarkb or others... | 21:00 |
| tmcpeak | I'll leave 0.17.1, shouldn't hurt anything | 21:00 |
| tmcpeak | should it? | 21:00 |
| * AJaeger has to leave now... | 21:00 | |
| *** salv-orlando has quit IRC | 21:00 | |
| tmcpeak | AJaeger: ok, sounds good, thanks for all the help! | 21:00 |
| AJaeger | clarkb: thanks! | 21:00 |
| *** daemontool has joined #openstack-infra | 21:00 | |
| *** daemontool_ has quit IRC | 21:03 | |
| tmcpeak | clarkb: ok so what's the best way to get out of this cleanly? | 21:05 |
| tmcpeak | if you have time please | 21:05 |
| tmcpeak | I'm lost :) | 21:05 |
| *** diazjf has joined #openstack-infra | 21:06 | |
| *** BobH_ has joined #openstack-infra | 21:06 | |
| clarkb | you have to run a change through code review, get it merged and tag that commit | 21:06 |
| *** derekh has joined #openstack-infra | 21:06 | |
| BobH_ | Greetings. Having an issue with a devstack I just got and running tempest. | 21:07 |
| clarkb | you cant push arbitrary commits to gerrit | 21:07 |
| BobH_ | Anyone see this while running run_tempst.sh -s: RuntimeError: no suitable implementation for this system, looks like monotonic is throwing an exception | 21:07 |
| *** daemontool_ has joined #openstack-infra | 21:07 | |
| jpr | greghaynes: I added another log of a build from a non-nodepool account. don't know why this makes a difference but it definitely succeeds. https://bugs.launchpad.net/diskimage-builder/+bug/1538135/+attachment/4557107/+files/dib-cli-image.log.gz | 21:07 |
| openstack | Launchpad bug 1538135 in diskimage-builder "disk-image-create deleting system /dev during image create " [Undecided,New] | 21:07 |
| tmcpeak | clarkb: ok, so I'll need to make a new branch for this? I can't just do a tag in master? | 21:07 |
| *** Sukhdev has quit IRC | 21:07 | |
| clarkb | you can do a tag on master, it just has to me merged from.gerrits perpsective | 21:08 |
| *** esker has joined #openstack-infra | 21:08 | |
| tmcpeak | clarkb: ok I have a bunch of commits that have happened from the time I'm trying to cherry pick against, so practically I'll probably need a branch, huh? | 21:09 |
| *** jsavak has quit IRC | 21:09 | |
| tmcpeak | clarkb: also I can't remove this tag I screwed up? | 21:09 |
| *** zz_ja has joined #openstack-infra | 21:09 | |
| jeblair | greghaynes, mordred, nibalizer: okay, i'm going to make the changes to nodepool.o.o now | 21:10 |
| *** jsavak has joined #openstack-infra | 21:10 | |
| *** aeng has joined #openstack-infra | 21:10 | |
| *** daemontool has quit IRC | 21:10 | |
| mordred | jeblair: I'm excited | 21:11 |
| *** ccrouch has joined #openstack-infra | 21:11 | |
| *** amrith is now known as _amrith_ | 21:11 | |
| jeblair | thanks to getting the secrets out of there, that basically just means copy it from project-config :) | 21:12 |
| mordred | jeblair: woot! | 21:12 |
| clarkb | tmcpeak: if you cant tag a current commit on master then yes | 21:13 |
| *** ashleighfarnham has quit IRC | 21:13 | |
| tmcpeak | clarkb: ok cool, fair enough | 21:13 |
| prometheanfire | mordred: works once I switch them around | 21:13 |
| *** piet has quit IRC | 21:13 | |
| *** esikachev has joined #openstack-infra | 21:14 | |
| jeblair | greghaynes: http://paste.openstack.org/show/485080/ | 21:14 |
| *** kgiusti has left #openstack-infra | 21:17 | |
| jeblair | greghaynes: (i rm-rfd that dir by hand to keep it moving) | 21:18 |
| AJaeger | tmcpeak: could you read http://docs.openstack.org/infra/manual/drivers.html and tell us if there's anything missing that would have avoided the problems you have, please? Changes are the best way to improve ... | 21:19 |
| * AJaeger really waves good-bye now | 21:19 | |
| *** sdake has quit IRC | 21:19 | |
| tmcpeak | AJaeger: ok, will do. thank you | 21:20 |
| prometheanfire | mordred: I'm going to submit this, let me know if you find anything, it's kinda large... http://paste.openstack.org/show/485082 | 21:20 |
| *** ashleighfarnham has joined #openstack-infra | 21:21 | |
| *** esikachev has quit IRC | 21:21 | |
| *** hdd has joined #openstack-infra | 21:23 | |
| *** boris-42 has quit IRC | 21:23 | |
| *** matrohon has quit IRC | 21:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Protect deleteNode from missing images https://review.openstack.org/272742 | 21:25 |
| *** matrohon has joined #openstack-infra | 21:26 | |
| *** bpokorny_ has quit IRC | 21:26 | |
| jeblair | mordred: NotFound: No Image matching {'name': 'CentOS 7'}. (HTTP 404) | 21:27 |
| *** achanda has joined #openstack-infra | 21:27 | |
| jeblair | mordred: (apparently that is not in bhs1) | 21:27 |
| *** bpokorny has joined #openstack-infra | 21:27 | |
| mordred | jeblair: piddle - I may have capsed wrong | 21:27 |
| *** rcernin has quit IRC | 21:28 | |
| mordred | jeblair: one sec | 21:28 |
| mordred | jeblair: p. "Centos 7" | 21:28 |
| mordred | jeblair: I'll send patch | 21:28 |
| *** bpokorny has quit IRC | 21:28 | |
| *** bpokorny has joined #openstack-infra | 21:29 | |
| jeblair | mordred: k, i've manually updated, will let you know if i see anything else | 21:29 |
| *** hdd has quit IRC | 21:29 | |
| *** salv-orlando has joined #openstack-infra | 21:30 | |
| *** achanda has quit IRC | 21:30 | |
| *** julim has quit IRC | 21:30 | |
| jeblair | weird. at one point nodepool could delete images without providers, but not any more | 21:30 |
| jeblair | (it still has the error messag saying it's going to delete an image with no provider) | 21:31 |
| *** tiswanso has quit IRC | 21:32 | |
| *** jcoufal_ has quit IRC | 21:32 | |
| *** tiswanso has joined #openstack-infra | 21:33 | |
| *** vgridnev has quit IRC | 21:33 | |
| *** Sukhdev has joined #openstack-infra | 21:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Change capitalization of CentOS 7 to Centos 7 https://review.openstack.org/272744 | 21:35 |
| *** baoli has joined #openstack-infra | 21:35 | |
| *** piet has joined #openstack-infra | 21:35 | |
| *** gtmanfred is now known as dontcare | 21:36 | |
| *** sridhar_ram has quit IRC | 21:36 | |
| *** trown is now known as trown|outttypeww | 21:37 | |
| ianw | mordred / jeblair : so thanks for getting these snapshot builds working. FYI i've been working on centos-minimal dib builds which should work everywhere. the issue with the dib images previously was that they weren't using glean and didn't do networking properly on rax. but that should be fixed now | 21:37 |
| jeblair | ianw: oh great | 21:37 |
| jeblair | ianw: (i don't know that we have it working, btw) | 21:37 |
| jeblair | ianw: we just wanted to try to have something in place for shutting down hpcloud | 21:38 |
| *** baoli_ has joined #openstack-infra | 21:38 | |
| ianw | yeah, mea culpa on not getting to this. i've been running with f23 and dropped ball on the hp retirement | 21:38 |
| jeblair | ianw: hopefully soon we'll have an answer about whether centos7 dib works on ovh | 21:38 |
| ianw | and sidetracked with things like pip breaking the world | 21:38 |
| ianw | it *should*, if dhcp works | 21:38 |
| jeblair | if so, we can drop the snapshot there, then look at rax | 21:38 |
| jeblair | ianw: i don't think the last 2 weeks have gone as anyone expected | 21:39 |
| *** jaosorior_ has quit IRC | 21:39 | |
| *** hashar has quit IRC | 21:40 | |
| *** baoli has quit IRC | 21:40 | |
| jeblair | mordred: our image lists are _much_ shorter now | 21:40 |
| *** baoli_ has quit IRC | 21:41 | |
| mordred | jeblair: yah - the 5-different-hps was quite multiplicative wasn't it? | 21:41 |
| *** baoli has joined #openstack-infra | 21:41 | |
| jeblair | yup | 21:41 |
| *** ayoung has quit IRC | 21:51 | |
| jeblair | okay, i think things are settling down | 21:51 |
| jeblair | hpcloud is only spinning up centos7 nodes, and the others are winding down | 21:51 |
| openstackgerrit | Merged openstack-infra/project-config: Change capitalization of CentOS 7 to Centos 7 https://review.openstack.org/272744 | 21:51 |
| *** alivigni has quit IRC | 21:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Fix typo in debug log https://review.openstack.org/272750 | 21:54 |
| nibalizer | jeblair: cool | 21:56 |
| *** e0ne has quit IRC | 21:57 | |
| *** ayoung has joined #openstack-infra | 21:58 | |
| *** yamahata has joined #openstack-infra | 21:59 | |
| *** yamahata has quit IRC | 22:01 | |
| *** sdake has joined #openstack-infra | 22:01 | |
| greghaynes | jeblair: ah, I forgbot about that delete bug with all the other breakage | 22:01 |
| *** yamahata has joined #openstack-infra | 22:01 | |
| *** ccrouch has quit IRC | 22:02 | |
| greghaynes | jpr: I dont understand - what change made your build succeed? | 22:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Clarify debug statements in image-upload https://review.openstack.org/272754 | 22:03 |
| tmcpeak | clarkb: still around? | 22:03 |
| clarkb | tmcpeak: sort of | 22:03 |
| jpr | greghaynes: i'm trying to figure that out. about to run the same commands from the nodepool account so I can diff | 22:03 |
| *** erikwilson has quit IRC | 22:03 | |
| greghaynes | jpr: ah, ok | 22:03 |
| tmcpeak | clarkb: how do I create a new branch for Bandit? | 22:03 |
| tmcpeak | I'd know how to do it if it wasn't gerrit controlled | 22:04 |
| *** ccrouch has joined #openstack-infra | 22:04 | |
| *** FallenPegasus has joined #openstack-infra | 22:04 | |
| *** tonytan4ever has quit IRC | 22:05 | |
| nibalizer | jeblair: i don't know if you saw, but we dropped tempest tests to 78 minutes with a 16vcpu omfra cloud node | 22:05 |
| jeblair | hrm, now it's trying to upload a dib image that has been deleted | 22:05 |
| jeblair | nibalizer: hrm, any idea what the bottleneck is? | 22:05 |
| clarkb | tmcpeak: looks like someone from the release team will have to do it based on your acl | 22:05 |
| tmcpeak | ok fair enough | 22:05 |
| *** gokrokve has quit IRC | 22:06 | |
| clarkb | tmcpeak: this should be documented in the infra manual let me get you a link | 22:06 |
| *** salv-orl_ has joined #openstack-infra | 22:06 | |
| tmcpeak | clarkb: thank you | 22:06 |
| nibalizer | jeblair: nope | 22:06 |
| nibalizer | but d-g is largely a black box to me | 22:06 |
| jeblair | nibalizer: you think the delay is in d-g? | 22:06 |
| *** regXboi has quit IRC | 22:06 | |
| tmcpeak | clarkb: I found it, thank you | 22:06 |
| *** diazjf has quit IRC | 22:07 | |
| nibalizer | jeblair: i mean my understanding is pretty limited | 22:07 |
| jeblair | nibalizer: if you want to separate job setup time from actual tempest runtime, tempest outputs the total time spent running tests at the end | 22:08 |
| nibalizer | I couldn't back of the hand walk you through what safe-devstack-vm-gate-wrap.sh does | 22:08 |
| *** Daisy has joined #openstack-infra | 22:08 | |
| jeblair | nibalizer: i'd start with that -- compare the runtime of tempest and see if the bulk of the extra time as compared to other providers is job setup or test execution | 22:08 |
| *** salv-orlando has quit IRC | 22:09 | |
| *** dprince has quit IRC | 22:09 | |
| nibalizer | ok | 22:10 |
| *** diazjf has joined #openstack-infra | 22:10 | |
| jeblair | nibalizer: (i'm assuming it's execution because the host is slow, but best to double check that first. also, double check that tempest ran 8 parallel processes (it's supposed to do #cpus/2)) | 22:10 |
| jeblair | nibalizer: (it outputs that info in the same place as the runtime) | 22:10 |
| *** bexelbie has quit IRC | 22:10 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 22:11 |
| greghaynes | jeblair: did you see ^ | 22:11 |
| jeblair | greghaynes: not yet | 22:11 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 22:11 |
| *** burgerk has quit IRC | 22:12 | |
| *** Daisy has quit IRC | 22:13 | |
| *** shashank_hegde has joined #openstack-infra | 22:14 | |
| jeblair | greghaynes, nibalizer, mordred, ianw: the centos7 dib image state on nodepool was missing the actual image files, so was unable to upload anything to ovh. i have deleted them and it just kicked off a dib build of centos7 | 22:15 |
| mordred | jeblair: neat | 22:15 |
| jeblair | ianw, mordred: i don't think we updated the logging config when we added the centos7 snapshot to ovh, so that's going to a black hole right now | 22:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Update nodepool log config https://review.openstack.org/272757 | 22:17 |
| *** jsavak has quit IRC | 22:17 | |
| jeblair | ianw, mordred: ^ | 22:17 |
| jeblair | nibalizer: i think clarkb said something about this change needing an update for a pidfile: https://review.openstack.org/271541 | 22:17 |
| *** dontcare is now known as gtmanfred | 22:17 | |
| *** sridhar_ram has joined #openstack-infra | 22:18 | |
| mordred | jeblair: crappit | 22:18 |
| jeblair | nibalizer: but he didn't leave a comment on the review, so i'm not positive | 22:18 |
| *** jsavak has joined #openstack-infra | 22:18 | |
| greghaynes | there was an issue where nodepool-builder and nodepoold attempted to sharea the same pid file | 22:18 |
| greghaynes | I have no idea whether that was resolved or not | 22:19 |
| *** matrohon has quit IRC | 22:19 | |
| nibalizer | jeblair: looking at it we set NAME to nodepool-builder and set PIDFILE to $NAME.pid | 22:19 |
| nibalizer | so I think we're ok | 22:19 |
| clarkb | nibalizer: jeblair the pidfile thing is fixed | 22:19 |
| clarkb | the current thing is we need to split the logfile confogs | 22:20 |
| jeblair | clarkb: has spoken! :) | 22:20 |
| nibalizer | clarkb: back to drinking you! /me swats | 22:20 |
| jeblair | clarkb: a comment in gerrit to that effect would be swell :) | 22:21 |
| *** sdake has quit IRC | 22:21 | |
| *** erikwilson has joined #openstack-infra | 22:21 | |
| ianw | jeblair: should be going to image.log | 22:23 |
| clarkb | done | 22:23 |
| jeblair | ianw: ah indeed thanks | 22:23 |
| jeblair | 2016-01-26 21:45:16,837 INFO nodepool.image.build.ovh-gra1.devstack-centos7: You are required to change your password immediately (password aged) | 22:23 |
| jeblair | 2016-01-26 21:45:16,842 INFO nodepool.image.build.ovh-gra1.devstack-centos7: Changing password for root. | 22:23 |
| *** erikwilson has quit IRC | 22:23 | |
| *** daemontool has joined #openstack-infra | 22:23 | |
| jeblair | ianw, mordred: i'm going to say that getting snapshots to work on ovh will pose a whole new set of challenges. | 22:24 |
| *** davidlenwell has quit IRC | 22:24 | |
| jeblair | ianw, mordred: i believe we should abandon them and focus only on dib | 22:24 |
| mordred | jeblair: I agree | 22:24 |
| *** ashleighfarnham has quit IRC | 22:24 | |
| *** calebb has quit IRC | 22:24 | |
| ianw | fair enough. with ovh, i haven't got an account yet as they require a prepayment on the credit card | 22:25 |
| *** davidlenwell has joined #openstack-infra | 22:25 | |
| jeblair | ianw: i will let you know the results of the dib build on ovh as soon as its ready | 22:25 |
| jeblair | ianw: do you think we should try uploading dib to rax now too? | 22:26 |
| ianw | jeblair: no, it won't work because it's not using glean/growroot etc. so it will try to dhcp and get no address | 22:26 |
| *** rlandy has quit IRC | 22:26 | |
| jeblair | ok | 22:26 |
| ianw | i think using centos-minimal is the way forward, but i've been having issues with that i'm working on | 22:26 |
| *** erlon has quit IRC | 22:26 | |
| jeblair | so fingers crossed for centos7-dib on ovh then? | 22:26 |
| *** derekh has quit IRC | 22:27 | |
| *** daemontool_ has quit IRC | 22:27 | |
| ianw | i think that should work, because it was working on hpcloud with dhcp | 22:27 |
| *** rlandy has joined #openstack-infra | 22:27 | |
| jeblair | k | 22:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Remove centos7 snapshots from ovh https://review.openstack.org/272761 | 22:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add flavor name filter to centos7-dib ovh https://review.openstack.org/272762 | 22:30 |
| jeblair | ianw, mordred, nibalizer: ^ | 22:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Further update nodepool log config https://review.openstack.org/272763 | 22:31 |
| *** EricGonc_ has quit IRC | 22:31 | |
| *** gokrokve has joined #openstack-infra | 22:31 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 22:32 |
| nibalizer | jeblair: +2, can +a if you have trouble finding one | 22:32 |
| *** sabeen3 has quit IRC | 22:35 | |
| *** jsavak has quit IRC | 22:35 | |
| zaro | nibalizer: maybe i don't need the logs, just some puppet help. | 22:36 |
| *** gokrokve has quit IRC | 22:36 | |
| nibalizer | zaro: sure how can I help | 22:36 |
| *** jsavak has joined #openstack-infra | 22:36 | |
| zaro | nibalizer: puppet report says this gets created on every puppet check, https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/init.pp#n451 | 22:36 |
| nibalizer | oh yea | 22:36 |
| *** sabeen has joined #openstack-infra | 22:36 | |
| nibalizer | one of the interns fixed that in the zuul mdule | 22:37 |
| zaro | trying to figure out why? | 22:37 |
| nibalizer | iirc | 22:37 |
| zaro | ref to fix? | 22:37 |
| nibalizer | https://review.openstack.org/#/c/198845/ | 22:37 |
| nibalizer | timothyb89: iirc | 22:37 |
| *** abregman has quit IRC | 22:37 | |
| zaro | did it fix the issue? | 22:38 |
| nibalizer | yes | 22:38 |
| *** sdake has joined #openstack-infra | 22:38 | |
| zaro | cool, thanks! | 22:38 |
| jpr | greghaynes: posted a log from the cli run of dib in the nodepool account that might help point to the issue. the differences start to appear around the point the image is populated, in the vicinity of the mv -t command | 22:39 |
| *** sabeen3 has joined #openstack-infra | 22:39 | |
| *** e0ne has joined #openstack-infra | 22:39 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 22:39 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Don't mistake manifest dir's for images https://review.openstack.org/272768 | 22:39 |
| greghaynes | jeblair: ^ fix for delete failures | 22:39 |
| *** tphummel has quit IRC | 22:39 | |
| *** daemontool_ has joined #openstack-infra | 22:39 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases https://review.openstack.org/272767 | 22:40 |
| jeblair | greghaynes: thx | 22:40 |
| *** tphummel has joined #openstack-infra | 22:40 | |
| *** ashleighfarnham has joined #openstack-infra | 22:40 | |
| *** sabeen has quit IRC | 22:41 | |
| *** dkranz has quit IRC | 22:42 | |
| *** daemontool has quit IRC | 22:42 | |
| *** thorst_ has joined #openstack-infra | 22:44 | |
| *** sdake has quit IRC | 22:46 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Tell puppet to check for cgid instead of cgi https://review.openstack.org/272769 | 22:47 |
| *** tonytan4ever has joined #openstack-infra | 22:47 | |
| *** tiswanso has quit IRC | 22:47 | |
| *** calebb has joined #openstack-infra | 22:47 | |
| *** thorst_ has quit IRC | 22:48 | |
| jeblair | greghaynes: http://paste.openstack.org/show/485087/ | 22:49 |
| openstackgerrit | Merged openstack-infra/nodepool: Protect deleteNode from missing images https://review.openstack.org/272742 | 22:49 |
| greghaynes | jpr: hrm, looked at the log, that one looks like it succeeded? | 22:49 |
| jeblair | greghaynes: that was the error that led me to delete the dib images in the first place; maybe there was no problem | 22:49 |
| *** ccrouch has quit IRC | 22:49 | |
| jeblair | other than whatever that bug is | 22:49 |
| openstackgerrit | Merged openstack-infra/project-config: Remove centos7 snapshots from ovh https://review.openstack.org/272761 | 22:50 |
| openstackgerrit | Merged openstack-infra/project-config: Add flavor name filter to centos7-dib ovh https://review.openstack.org/272762 | 22:50 |
| jpr | greghaynes: the one prefixed with nodepool- took my system /dev away | 22:50 |
| *** sdake has joined #openstack-infra | 22:50 | |
| *** amitgandhinz has quit IRC | 22:50 | |
| greghaynes | jpr: oh, I just have dib-cli-image.log, where is the nodepool- one? | 22:51 |
| zaro | clarkb: that reply to jenkins bug was pretty confusing, you should just make nodepool use the entire python-jenkins library. | 22:52 |
| *** weshay has quit IRC | 22:52 | |
| jpr | greghaynes: hmm, I've attached a dib-cli-image-2.log and nodepool-dib-cli-image.log to the ticket. | 22:53 |
| jpr | will follow up posts | 22:53 |
| *** tmcpeak has left #openstack-infra | 22:53 | |
| jpr | the last two are the identical commands save for the paths for the accounts | 22:54 |
| jpr | *with follow up posts | 22:54 |
| *** boris-42 has joined #openstack-infra | 22:54 | |
| greghaynes | jpr: ah, I see now | 22:54 |
| *** dims has joined #openstack-infra | 22:55 | |
| openstackgerrit | Doug Fish proposed openstack-infra/project-config: Add JavaScript gate jobs to LBaaS dashboard https://review.openstack.org/272771 | 22:55 |
| greghaynes | jpr: I have one more request if you have a bit of time | 22:55 |
| greghaynes | jpr: Can you modify the disk-image-create executable and add a set -x at the top of it, then rerun nodepool and grab that log | 22:56 |
| greghaynes | jpr: that should add a ton of debug info | 22:56 |
| jpr | sure. working on a parallel effort so I may bounce a bit | 22:56 |
| jpr | ok | 22:56 |
| greghaynes | ok | 22:56 |
| zaro | clarkb: gerrit add reviewer takes forever to suggest your name for some reason. | 22:57 |
| *** ccrouch has joined #openstack-infra | 22:57 | |
| *** achanda has joined #openstack-infra | 22:58 | |
| *** tonytan4ever has quit IRC | 22:59 | |
| *** doug-fish has quit IRC | 23:00 | |
| *** sdake has quit IRC | 23:01 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 23:01 |
| *** e0ne has quit IRC | 23:02 | |
| jeblair | greghaynes: ah, that error is because we are looking for a raw but we only have qcow2 | 23:02 |
| greghaynes | jeblair: which error? | 23:03 |
| greghaynes | so many errors today | 23:03 |
| jeblair | BuilderInvalidCommandError: Unable to find image file for id 176212 to upload | 23:03 |
| greghaynes | ah, makes sense | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add more debug logging to image upload https://review.openstack.org/272775 | 23:06 |
| jeblair | logging i added to discover that ^ | 23:07 |
| jeblair | okay, so now -- why isn't there a raw image there? | 23:07 |
| greghaynes | jeblair: who consumes raw images? | 23:08 |
| jeblair | greghaynes: ovh apparently | 23:09 |
| nibalizer | mordred: I think puppetdb is ooming | 23:09 |
| greghaynes | hrm | 23:09 |
| nibalizer | is why it keeps restarting | 23:09 |
| greghaynes | and we didnt add that provider recently, correct? There is an issue where if you build before you add the provider, then you try to upload the image it will not have built all the needed image types | 23:09 |
| nibalizer | JAVA_ARGS="-Xmx192m that makes sense | 23:10 |
| greghaynes | I have to afk for a bit | 23:10 |
| jeblair | greghaynes: this image build is brand new, after the provider (and the image was added to that provider) | 23:10 |
| nibalizer | thats the java I know | 23:10 |
| jeblair | greghaynes: however... | 23:10 |
| jeblair | greghaynes: is that in the builder now? | 23:10 |
| jeblair | greghaynes: and does the builder ever reload the config? | 23:10 |
| jeblair | greghaynes: perhaps the builder was out of sync with the nodepool config | 23:11 |
| greghaynes | jeblair: oh. it doesnt reload config | 23:11 |
| jeblair | whoops | 23:11 |
| *** tongli has quit IRC | 23:11 | |
| greghaynes | that thought had never occured to me | 23:11 |
| greghaynes | ok, so new feature to work on | 23:11 |
| jeblair | okay, i'm not going to fix that now. i'm going to rely on the fact that i have since restarted the builder, and then delete that dib image and let it build it again | 23:11 |
| greghaynes | Yep, I can poke at that when I return | 23:11 |
| *** baoli has quit IRC | 23:12 | |
| *** baoli has joined #openstack-infra | 23:13 | |
| *** sdake has joined #openstack-infra | 23:13 | |
| *** tongli has joined #openstack-infra | 23:13 | |
| jeblair | 2016-01-26 23:14:26,108 INFO nodepool.builder: Running disk-image-create -x -t raw --no-tmpfs -o /opt/nodepool_dib/176213 centos7 vm openstack-repos nodepool-base node-devstack cache-bindep | 23:14 |
| jeblair | looks better | 23:14 |
| *** zz_dimtruck is now known as dimtruck | 23:14 | |
| *** Sukhdev has quit IRC | 23:17 | |
| *** jsavak has quit IRC | 23:19 | |
| *** notnownikki has quit IRC | 23:21 | |
| openstackgerrit | Merged openstack/requirements: update to oslo.config 3.4.0 https://review.openstack.org/272607 | 23:21 |
| *** diazjf has quit IRC | 23:22 | |
| *** calebb has quit IRC | 23:24 | |
| ianw | jeblair / clarkb : all git hosts now centos7 right? | 23:25 |
| ianw | just looking at some system-config bits and old work-arounds we had in there | 23:25 |
| *** gokrokve has joined #openstack-infra | 23:28 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:28 | |
| *** darrenc is now known as darrenc_afk | 23:28 | |
| *** derekh has joined #openstack-infra | 23:29 | |
| *** sdake has quit IRC | 23:31 | |
| *** gokrokve has quit IRC | 23:33 | |
| *** jcoufal has joined #openstack-infra | 23:33 | |
| *** darrenc_afk is now known as darrenc | 23:36 | |
| *** dimtruck is now known as zz_dimtruck | 23:37 | |
| *** ganso has quit IRC | 23:39 | |
| *** sabeen3 has quit IRC | 23:39 | |
| *** piet has quit IRC | 23:39 | |
| jpr | greghaynes: i've added the log with additional debug | 23:41 |
| jeblair | ianw: yes | 23:41 |
| jeblair | ianw: we're uploading the centos7-dib images to ovh now | 23:42 |
| *** ashleighfarnham has quit IRC | 23:43 | |
| *** rajinir has quit IRC | 23:45 | |
| *** nelsnelson has quit IRC | 23:47 | |
| *** nelsnelson has joined #openstack-infra | 23:47 | |
| *** zz_dimtruck is now known as dimtruck | 23:47 | |
| *** rbrndt_ has quit IRC | 23:49 | |
| *** rbrndt has joined #openstack-infra | 23:50 | |
| *** Qiming has joined #openstack-infra | 23:50 | |
| *** vponomaryov has quit IRC | 23:52 | |
| *** vponomaryov has joined #openstack-infra | 23:53 | |
| *** flepied has quit IRC | 23:54 | |
| *** diana_clarke has joined #openstack-infra | 23:54 | |
| openstackgerrit | Shu Muto proposed openstack/os-testr: remove python 3.3 trove classifier https://review.openstack.org/264521 | 23:55 |
| *** ashleighfarnham has joined #openstack-infra | 23:57 | |
| *** yamamoto_ has joined #openstack-infra | 23:58 | |
| *** austin81 has left #openstack-infra | 23:58 | |
| *** achanda has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!