Thursday, 2015-03-12

*** annegentle has joined #openstack-swift00:06
*** dmorita has joined #openstack-swift00:07
*** edmondk has joined #openstack-swift00:21
edmondkQuick question: Does anyone here know what the /opt/stack/data/swift/drives/images/swift.img file is for and does it ever clean itself up after removing swift elements?00:23
claygprobably never shrinks - its the file backed loop back device with all the devstack swift devices00:23
notmynameedmondk: is that something from devstack? (it's not an artifact created by code in the swift repo)00:23
edmondkRight now after uploading images to glance and creating swift elements and deleting my hard drive is still full even though no images exist00:24
claygthere's probably some devstack way to say resetswift00:24
edmondknotmmyname: yeah using devstack00:24
*** bkopilov has quit IRC00:24
edmondkdoes it hurt to just delete the file?00:24
InAnimaTeso...python-swiftclint....does it support 307's?00:24
claygedmondk: it would probably only break your dev swift setup00:24
InAnimaTe(basically just had this discussion in #haproxy: http://tty0.in/view/c656172a)00:24
notmynameclayg: oh, is that a loopback mount fro swift in devstack?00:24
edmondkit has essentially filled the entire volume so I can't upload anymore images00:24
claygedmondk: I get it - that sucks - you're going to have to delete it and recreate it - there's probably a way to tell devstack to do that - I don't know what it is00:25
claygchmouel: acoles_away: cschwede: you guys know how to spell resetswift on devstack for edmondk00:25
notmynameInAnimaTe: unlikely. swift doesn't generate any 3xx responses.  .....hmmm... /me wonders if the requests lib does it automatically00:26
chmoueledmondk:  ./unstack.sh00:26
chmouel./stack.sh00:26
InAnimaTeok, im asing since im interested in using haproxy merely to help the client make a decision on which node to talk to, but i dont want any actual PUT's or GET's (or head's technically) going through the proxy, i just want it to be a broker00:27
claygInAnimaTe: there's a couple of conversations going on in that thread - it sounds like you're going to make haproxy generate the 3XX - and if maybe swiftclient will fall for it?00:27
InAnimaTeyep00:27
InAnimaTethats the goal00:27
edmondkYeah figured unstack and stack would fix it, just without having to do that would be ideal because I have a bunch of instances running that were pretty hard to get setup right00:27
claygInAnimaTe: i'm into it - probably won't work out of the box - have fun hacking - keep us posted00:28
claygchmouel: thanks00:29
claygedmondk: go digging in the devstack code - there's some lib function that makes that file - you could probably cook something up00:29
edmondkk00:29
claygedmondk: yeah my money is on delete the file and unmount it all that jazz by hand - then try and run create_swift_disk in devstack/lib/swift00:30
*** gyee has quit IRC00:31
*** annegentle has quit IRC00:31
openstackgerritYuan Zhou proposed openstack/swift: List valid EC types in error  https://review.openstack.org/14231100:32
edmondkyeah I am in that func in devstack/lib/swift now00:32
claygyuanz: doh!  "on"00:32
claygedmondk: sorry we couldn't be more help - most of us only use devstack when we have to test something with keystone00:32
edmondkclayg It looks like if I just call that method it checks for the directory and if found does a sudo unmount and sudo rm on it00:32
claygphthhtppthtphtph - EVEN BETTER!00:33
edmondkclayg: thanks for the help will give that a go, if it fails I will go with the nuclear option and unstack and stack00:33
tsg_torgomatic: jenkins is happy (https://review.openstack.org/#/c/163533/) is there a way to trigger "cluster-swift-tox-func"?00:35
*** annegentle has joined #openstack-swift00:38
*** vishy has quit IRC00:41
*** vishy has joined #openstack-swift00:41
*** zhill has quit IRC00:44
*** tsg_ has quit IRC00:47
*** annegentle has quit IRC00:48
*** annegentle has joined #openstack-swift00:49
*** chlong has quit IRC00:55
*** chlong has joined #openstack-swift00:55
*** annegentle has quit IRC01:08
edmondkclayg: I unmounted the file removed it, created a new swift.img and restarted swift but am unable to create containers now. I get Could not autocreate account '/AUTH_67bec053e00d4a24b205b083ccae7051' on s-proxy01:09
edmondkclayg: I am just going to go ahead and unstack and stack01:10
claygyeah the swift.img needs to be loopbacked, partitioned and have filesystems on them, and mounted into the right places - it's a chore to do by hand and devstack knows how to do it somewhere01:11
claygi thought getting that function to run might have put things back in the right place01:11
claygnext time I have my devstack box up I'll try and remember to take stab at it - put something in the interwebz for the next guy01:11
pelusetorgomatic, clayg :  tsg's pyeclib patch is ready for 2nd review (so EC patches can pass Jenkins)... https://review.openstack.org/#/c/163533/01:13
peluseyuanz, thanks again for jumping back on the EC stuff to help our beta push!01:13
*** clds_ has quit IRC01:42
*** clds has joined #openstack-swift01:44
yuanzpeluse, no problem, it's my honor :)01:55
*** haomaiwang has joined #openstack-swift02:22
*** david-lyle is now known as david-lyle_afk02:45
*** echevemaster has quit IRC02:53
*** bkopilov has joined #openstack-swift02:58
notmynamedfg: redbo: I'd like for https://review.openstack.org/#/c/126923/ to land. please take a look if you're interested (fsync() on dirs)03:15
*** bkopilov has quit IRC03:19
clayghugokuo: charz: the qa cluster is real hardware right - can we add a ssbench run to the checks?03:26
notmynameclayg: yes, and probably. might need to rune them from the jenkins node03:26
notmynameclayg: what do you want to see? a raw number or comparison over time?03:26
claygoh does the swiftqa fabric scripts not control what acctually gets run03:27
notmynamefabric like .fab files? is that how charz set it up?03:27
notmynameI think it's the jenkins gerrit plugin03:28
claygi saw something in console.log for github.com/swiftstack/swiftqa03:28
notmynamebut I don't really know03:28
claygfab start_services03:29
charzclayg: yeah, I use fabric to do that.03:29
claygso but who is acctually deciding to run tox to kick off the func tests?  is that something you have to co-ordinate with infra?03:30
claygor do they just like - here's a new change - do your thing03:30
notmynameclayg: the jenkins gerrit plugin is sniffing all the events from gerrit. and it acts on what it wants03:31
charzclayg: jenkins gerrit plugin did.03:31
claygok, so we have our own jenkins server, and we can make any jobs we want - then they get triggered by the gerrit hook plugin - i love it03:32
claygso yeah - let's add an ssbench run job :D03:32
notmynameclayg: `ssh -p 29418 USERNAME@review.openstack.org gerrit stream-events`03:32
claygall that jazz is all up in your anisble's anyway - doesn't have to be anything fancy03:32
charzclayg: jenkins server 192.168.201.155:808003:33
clayghell yeah03:33
notmynameclayg: incoming....03:35
charzclayg: notmyname How about add 1k/1m/10m PUT/GET benchmark? or we need more?03:37
notmynamecharz: just for fun, I'd say run it with deletes turned off. at least for a while. that will gradually fill up the cluster03:38
claygcharz: that'd be fine - but it looks like it'd be easier to just add it to the existing job than to create a new job?03:38
claygnotmyname: there's a clean script that wipes disk03:38
notmynamenew job, please03:38
notmynameclayg: I don't /think/ that's being run03:38
notmynamecharz: ?03:38
claygnotmyname: I *just* said it'd be *easier* to add it the existing job - and you *immediately* ask for a new job :P03:39
notmynamelol03:39
notmynamein my experience, it's trivial to add a new job. at the end of the day, I mention it in passing to charz, and it's there by the time I wake up ;-)03:40
clayg+1 charz is amazing03:41
charznotmyname: clayg I'll create a new job for it.03:44
notmynamethanks. makes it easier to track and manage03:44
*** ppai has joined #openstack-swift03:44
mattoliveraunotmyname, clayg: So from what I read from your conversation, charz is like a cross between a  magic genie and tooth fairy? you wish for something, go to sleep and when you wake up its done :P04:01
claygpeluse: acoles_away: ok, i'm going to push up a rebase04:07
charzmattoliverau: like a ninja. :P04:07
claygi tried to squash all the policy cleanup into multi-fragment but it was super obvious writing the commit the patch was just trying to do too many things at once04:08
claygso I rebased policy-everywhere in front of the suffix changes (I think it made the suffix change simpler)04:08
clayganyway - if it doesn't work out - that's why we use version control right!04:08
claygcharz: *thats* it!  a *ninja* - i was trying to find the right word...04:09
mattoliveraulol04:10
charzLoL04:10
openstackgerritClay Gerrard proposed openstack/swift: Multiple fragment Archive Index Support  https://review.openstack.org/15963704:12
openstackgerritClay Gerrard proposed openstack/swift: Prefer policy to policy_index  https://review.openstack.org/16369004:12
mattoliverausigh.. just as I was in the middle of reviewing these :P04:12
claygmattoliverau: https://review.openstack.org/#/c/163690/1 will be easy04:14
mattoliverauclayg: famous last words :P04:16
*** devlaps has quit IRC04:25
openstackgerritClay Gerrard proposed openstack/swift: Multiple fragment Archive Index Support  https://review.openstack.org/15963704:29
*** erlon has quit IRC04:41
*** bkopilov has joined #openstack-swift04:56
*** david-ly_ has joined #openstack-swift05:04
*** yuanzz has joined #openstack-swift05:04
*** jd__` has joined #openstack-swift05:05
*** thurloat_ has joined #openstack-swift05:09
*** yuanz has quit IRC05:09
*** david-lyle_afk has quit IRC05:09
*** cebruns has quit IRC05:09
*** jd__ has quit IRC05:09
*** mariusv has quit IRC05:09
*** thurloat has quit IRC05:09
*** ho__ has quit IRC05:09
*** thurloat_ is now known as thurloat05:09
*** jd__` is now known as jd__05:09
*** BAKfr has quit IRC05:09
*** redbo has quit IRC05:09
*** ho__ has joined #openstack-swift05:09
*** mariusv has joined #openstack-swift05:09
*** mariusv has quit IRC05:09
*** mariusv has joined #openstack-swift05:09
*** cebruns has joined #openstack-swift05:09
*** redbo has joined #openstack-swift05:09
*** ChanServ sets mode: +v redbo05:09
*** BAKfr has joined #openstack-swift05:13
*** reed has quit IRC05:30
*** SkyRocknRoll has joined #openstack-swift06:04
*** mmcardle has joined #openstack-swift06:20
*** silor has joined #openstack-swift06:21
*** mmcardle has quit IRC06:26
*** nshaikh has joined #openstack-swift06:29
*** ho__ has quit IRC06:46
*** ho has quit IRC07:14
*** silor has quit IRC07:16
*** SkyRocknRoll has quit IRC07:31
*** sluo_wfh has joined #openstack-swift07:42
*** sluo_wfh has quit IRC07:53
*** jistr has joined #openstack-swift08:03
*** sluo_wfh has joined #openstack-swift08:18
*** geaaru has joined #openstack-swift08:24
*** rledisez has joined #openstack-swift08:32
*** jordanP has joined #openstack-swift08:33
*** acoles_away is now known as acoles08:56
*** nellysmitt has joined #openstack-swift09:02
*** mmcardle has joined #openstack-swift09:30
*** SkyRocknRoll has joined #openstack-swift09:51
*** jordanP has quit IRC10:00
*** BAKfr has quit IRC10:35
*** BAKfr has joined #openstack-swift10:35
*** jistr has quit IRC10:36
m_hanhey guys ! i'm having an issue, my i'cant start my proxy-server : "swift-init main start gives the following exception http://paste.openstack.org/show/191794/10:38
m_hanthis is the proxy-server.conf http://paste.openstack.org/show/191795/  any suggestions? thanx & best regards10:40
*** BAKfr has quit IRC10:44
*** BAKfr has joined #openstack-swift10:44
*** haomaiwang has quit IRC10:48
omamem_han: port already in use?10:51
*** jistr has joined #openstack-swift10:53
m_hanomame:  yes, it is listenning on tcp, but i have neither tomcat or apache running on my machine10:57
omamesudo netstat -ntlp | grep :8080 should give you a hint10:58
omamemaybe it's proxy-server itself :)10:58
m_hantcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      1804/python10:58
omamecause you're starting, not restarting10:58
omametry swift-init proxy-server restart10:58
*** erlon has joined #openstack-swift10:58
m_hani get the same error http://paste.openstack.org/show/191797/11:01
omamewho's process 1804?11:02
m_hanit was /usr/bin/python /usr/local/bin/swift-proxy-server /etc/swift/proxy-server.conf11:03
m_hani killed it and it works fine now xD11:03
omame;)11:03
m_hanthank u for ur support ;)11:04
omameno problem11:05
openstackgerritDaisuke Morita proposed openstack/swift: object-replicator warns when object dirs mismatch with policy settings  https://review.openstack.org/16338911:08
*** jrichli has joined #openstack-swift11:17
*** silor has joined #openstack-swift11:18
*** aix has joined #openstack-swift11:19
*** km has quit IRC11:20
*** nellysmi_ has joined #openstack-swift11:24
*** nellysmitt has quit IRC11:26
*** mahatic has joined #openstack-swift11:39
openstackgerritDenis Cavalcante proposed openstack/swift: Add storage policy support for sorting method  https://review.openstack.org/16087711:39
*** nshaikh has quit IRC11:43
*** dmorita has quit IRC11:48
*** mahatic has quit IRC11:51
*** aix has quit IRC11:59
*** tsg_ has joined #openstack-swift11:59
*** mahatic has joined #openstack-swift12:06
*** mahatic has quit IRC12:13
*** mahatic has joined #openstack-swift12:15
*** silor has quit IRC12:25
*** SkyRocknRoll has quit IRC12:25
*** panbalag has joined #openstack-swift12:34
*** SkyRocknRoll has joined #openstack-swift12:39
*** mmcardle has quit IRC12:41
*** ahale is now known as _ahale12:49
*** NM has joined #openstack-swift12:50
*** jkugel has joined #openstack-swift12:57
*** aix has joined #openstack-swift12:58
*** bkopilov has quit IRC13:01
acolestsg_: hi you here?13:02
tsg_acoles: yes, how is it going13:02
acolestsg_: good, thanks for trying those rechecks - saw your note about rebasing, i will look at that again later today. thx13:03
*** panbalag has quit IRC13:05
straycatWhy do openstack things tend to have bind_ip rather than bind_interface?13:06
straycate.g. proxy-server.conf13:06
tsg_acoles: yes, this exercise has been a little troublesome :) given we only have ubuntu packages built for the libs so far, we have had to get a special version of pyeclib up on pypi.  Jenkins was unhappy yesterday because of some venv related issues that caused the early failures.  Those were fixed later in the afternnon PDT and pyeclib 1.0.3 version should work just fine now.13:06
acolestsg_: ok thx13:06
tsg_acoles: clayg was having some issues last night - once those are resolved and he clears https://review.openstack.org/#/c/163533, we should be good for a rebase13:08
tsg_acoles: would like to hear from you if you see any issues after you update pyeclib (and liberasurecode by proxy)13:08
tsg_acoles: would like to get any caveats noted in the pyeclib README13:09
*** SkyRocknRoll has quit IRC13:09
*** SkyRocknRoll has joined #openstack-swift13:10
*** mmcardle has joined #openstack-swift13:13
*** bill_az has joined #openstack-swift13:20
*** mmcardle has quit IRC13:24
*** mmcardle has joined #openstack-swift13:26
*** annegentle has joined #openstack-swift13:37
acolestsg_: so do we need to update requirements.tx for pyeclib>=1.0.3 ?13:41
tsg_acoles: that's correct13:41
tsg_acoles: I need to get that change into global-requirements and then on swift master13:42
acolestsg_: ok ic13:42
tsg_acoles: waiting on some feedback today to make sure we don't need a 1.0.4 :)13:43
tsg_I don't anticipate changes in the core library, only some documentation to clarify dependencies better, etc13:45
*** jrichli has quit IRC13:53
acolestsg_: so i pulled patch 163533, pip installed pyeclib 1.0.3, looks like i got liberasurecode 1.0.4, but see exception in unit test http://paste.openstack.org/show/191836/ - any ideas?13:55
patchbotacoles: https://review.openstack.org/#/c/163533/13:55
*** jrichli has joined #openstack-swift13:55
openstackgerritPrashanth Pai proposed openstack/swift: Refactor server side copy as middleware  https://review.openstack.org/15692313:56
tsg_acoles: does /var/log/syslog show anything13:58
acolestsg_: ah! Mar 12 13:51:36 localhost liberasurecode[11835]: liberasurecode_backend_open: dynamic linking error libgf_complete.so.1: cannot open shared object file: No such file or directory13:59
tsg_acoles: we are trying to get rid of this limitation (syslog only logging) but given liberasurecode is a C library, we couldn't easily unify the logs :)14:00
tsg_that apart, do you have libgf_complete installed under /usr/local/lib?14:00
tsg_acoles: perhaps run ldconfig?14:00
* tsg_ adding these notes to a "troubleshooting" section in pyeclib docs14:01
*** Fin1te has joined #openstack-swift14:02
acolestsg_: i have libgf_complete.so.1.0.0 , ldconfig seems to have done the trick, just running complete suite of unit test14:02
tsg_great14:03
*** ppai has quit IRC14:03
tsg_acoles: I am thinking that may have been clayg's problem as well :)14:04
acolestsg_: ok all unit tests pass now. thanks!14:05
tsg_acoles: good to know!  I will get this documented14:05
acolestsg_: i think (can't be sure) i had symlink to libgf_complete.so.1.0.0 before running ldconfig.14:05
tsg_acoles: given we use autoconf+libtool for those libs, you shouldn't have to, but I will double check14:10
*** Fin1te has quit IRC14:14
*** Fin1te has joined #openstack-swift14:14
*** Fin1te has quit IRC14:15
acolestsg_: so looks like maybe patch 163690 needs to be rebased onto 163533 and then the chain above that will pick up this fix?14:16
patchbotacoles: https://review.openstack.org/#/c/163690/14:16
tsg_acoles: that's correct14:18
acolestsg_: ok i will try that14:18
acolespeluse: you awake yet?14:19
tsg_acoles: peluse may be on a plane to SF14:19
acolestsg_: he's awake then :)14:19
tsg_acoles: :) heh, yes14:20
acolestsg_: iirc he has a private jet ;)14:21
tsg_acoles: yeah ;) and he is nice enough to lend it to a company named "Intel" sometimes14:22
tsg_heh14:22
acolestsg_: lol. he's good to them14:23
*** lcurtis has joined #openstack-swift14:43
*** SkyRocknRoll has quit IRC14:58
*** csmart has quit IRC15:03
*** csmart has joined #openstack-swift15:04
*** rdaly2 has joined #openstack-swift15:04
*** bkopilov has joined #openstack-swift15:07
*** lcurtis has quit IRC15:10
openstackgerritAlistair Coles proposed openstack/swift: Prefer policy to policy_index  https://review.openstack.org/16369015:23
*** david-ly_ is now known as david-lyle15:23
acolesclayg: ^^ fixed pep8 issues and rebased on patch 163533 to try to get tests to pass, hope thats ok!15:24
patchbotacoles: https://review.openstack.org/#/c/163533/15:24
*** nellysmi_ has quit IRC15:24
tsg_acoles, thanks for the note on tox.  btw, there is also an eventlet update sitting in the queue https://review.openstack.org/#/c/145403/15:25
*** thumpba has joined #openstack-swift15:26
tsg_acoles, jenkins hasn't been happy with these given these changes are submitted against feature/ec and there is no such branch in the global requirements repo15:27
tsg_acoles, will need to get these updates submitted to swift master15:27
acolesnotmyname: i noticed that openstackgerrit is not reporting feature/crypto reviews in this channel - who should i talk to about fixing that?15:29
*** thumpba_ has joined #openstack-swift15:32
*** reed has joined #openstack-swift15:34
*** thumpba has quit IRC15:35
tsg_acoles: I am going to work on making pyeclib/liberasurecode logging configurable so we can control where those messages go from swift15:39
tsg_acoles: please do let me know any suggestions you may have15:39
*** zul has quit IRC15:40
*** zul has joined #openstack-swift15:41
*** nermik has joined #openstack-swift15:57
nermikHi! newbie question - does swift support partial file retrieval? if I define byte range for media file..15:59
acolesnermik: yes with a Range header http://developer.openstack.org/api-ref-objectstorage-v1.html#storage_object_services16:02
nermikand if media is chunked and I need data from number of chunks - should I make number of calls and then "glue" chunks together ?16:08
nermikthere is still 5GB object limitation ?16:08
openstackgerritLorcan Browne proposed openstack/swift: Add swift-recon feature to track swift-drive-audit error count  https://review.openstack.org/16388916:09
ctennisyes and yes and http://docs.openstack.org/developer/swift/overview_large_objects.html16:09
*** thumpba has joined #openstack-swift16:12
nermikctennis, acoles: thanks!16:12
ctennisyou're welcome!16:13
*** dencaval has joined #openstack-swift16:14
*** bkopilov has quit IRC16:15
*** thumpba_ has quit IRC16:16
*** tsg_ has quit IRC16:17
*** bkopilov has joined #openstack-swift16:17
*** bkopilov has quit IRC16:20
*** thumpba_ has joined #openstack-swift16:21
m_hanhi, how to delete an object from swift-cluster ?16:23
*** thumpba has quit IRC16:23
ctennism_han: issue an HTTP DELETE to the object16:24
m_hanctennis: it says http://paste.openstack.org/show/191870/16:27
ctennisI don't think you're authenticated, you have to also pass your auth token as part of the request16:27
ctennisalso that doesn't look like a valid url16:28
ctennisthe /endpoints isn't right16:28
*** bsdkurt has joined #openstack-swift16:28
m_hanthe url is fine http://paste.openstack.org/show/191872/16:35
m_hanmay be i'm missing another option in the curl command16:35
m_hanany ideas?16:36
*** jistr has quit IRC16:45
ctennisno, the endpoint url you are using is hitting a different API16:47
ctennisif you were doing an actual GET on the object, it would return the contents and not the endpoints16:47
ctennisthe object url is /v1/account/container/object16:47
*** kirubak has joined #openstack-swift16:54
*** edmondk has left #openstack-swift16:55
notmynamegood morning17:19
*** Nadeem has joined #openstack-swift17:22
*** aix has quit IRC17:25
*** rledisez has quit IRC17:26
*** devlaps has joined #openstack-swift17:26
*** kirubak has quit IRC17:32
*** nermik has quit IRC17:36
jrichlinotmyname: morning.  good trip?17:50
notmynameyup. it was great to see people17:51
notmynameI like the ops summits. the sysadmins get an opportunity to talk and say what they really think without a lot of the traditional loud voices in openstack drowning them out17:51
jrichli:)17:52
notmynameand I was told on 2 different occasions this week that what the ops people are asking for is generally stuff that swift has always done. so that's nice :-)17:53
jrichliawesome!17:53
*** geaaru has quit IRC17:54
*** zhill has joined #openstack-swift17:59
*** annegentle has quit IRC18:00
*** annegentle has joined #openstack-swift18:12
*** annegentle has quit IRC18:15
*** annegentle has joined #openstack-swift18:15
*** bill_az has quit IRC18:21
*** bkopilov has joined #openstack-swift18:21
*** tsg_ has joined #openstack-swift18:23
*** fbo has quit IRC18:23
notmynameacoles: jrichli: I'll look in to getting the feature/crypto branch known by the patch bot18:24
jrichlithanks!18:25
acolesnotmyname: thx18:25
* notmyname remembers something about a yaml file being updated18:25
*** nermik has joined #openstack-swift18:26
*** fbo has joined #openstack-swift18:26
notmynameyup18:27
*** fbo has quit IRC18:30
notmynameacoles: jrichli: https://review.openstack.org/#/c/163927/18:30
claygtsg_: torgomatic: acoles: yuanzz: so `sudo ldconfig` fixed things for me - i already had 1.0.4 liberasure installed?18:31
torgomaticclayg: that's good18:31
acolesclayg: same for me18:32
torgomaticI imagine this whole thing will get easier once pyeclib gets version checking installed18:32
tsg_pyeclib does have version checking in fact18:32
tsg_so it would install the right version (update if necessary)18:32
tsg_in clayg's case, it must have been just the missing ldconfig18:32
torgomaticwell, there you go :)18:33
tsg_torgomatic, clayg: liberasurecode is configured to send error messages to syslog by default18:33
tsg_I am trying to change that to also log to stderr, do you have a suggestion there?18:34
claygtsg_: so the user having to run ldconfig is expected?18:34
torgomatictsg_: wait, so under my Python code is a Python library and then under that is a C library that talks to my syslog?18:34
claygtsg_: torgomatic: did all of you just *know* that and I'm some sort of johnny?18:34
*** fbo has joined #openstack-swift18:34
torgomaticclayg: the ldconfig thing? it said it on my terminal after the upgrade, so I did it18:35
claygheh - i don't read that shit ;)18:35
notmynameacoles: jrichli: and it's already approved. as soon as it lands, the bot will auto-restart and it will be ready18:35
torgomaticclayg: evidently ;)18:35
*** nellysmitt has joined #openstack-swift18:35
jrichlinotmyname: great!  now if only I will be ready to upload my latest in a day or so :)18:37
tsg_torgomatic: on the syslog comment, yes .. liberasurecode (C) is talking to syslog (we didn't change the defaults, but can do so)18:37
claygacoles: so per policy diskfiles - I'm down for all the same reasons I think ECObjectController was a good idea18:37
tsg_torgomatic: what would be the ideal way? stderr? or configure both pyeclib/liberasurecode to swift logs?18:38
claygacoles: I think i want to add an distutils entrypoint for storage policy classes - and have them register their objectcontroller and diskfile in their factory method18:38
claygacoles: but before we need that we can just have the ECStoragePolicy sort of do it in tree18:39
acolesclayg: are we talking objectcontroller as in backend server or proxy controller class?18:40
claygI dream of a merge to master that is almost entirely *additive* like - here's the controller, and the diskfile, and it's off in it's own little sandbox, and very little shared code has to change except for where we add the routing hooks to lookup the controller and diskfile based on the policy18:40
claygacoles: yeah if you look in feature/ec's proxy.controllers.obj - there's already a ECObjectController class18:40
acolesclayg: yeah, so are you also thinking ECObjectController in swift.obj.server ?18:41
acolesam i confused?18:41
*** zul has quit IRC18:41
claygacoles: oh... i wasn't thinking about that :\18:41
*** fbo has quit IRC18:41
claygThe difference with the storage server controllers and the proxy controllers is that a proxy controller instance gets created for every request18:41
acolesok i *was* confused :)18:41
acolesclayg: yeah, ic18:42
claygacoles: so part of the change was that I was thinking a single object server could service requests for different diskfiles18:42
acolesclayg: yes18:43
claygso there'd have to be a "router" step - probably in object.server.ObjectController.get_diskfile18:43
*** zul has joined #openstack-swift18:43
acolesclayg: yup18:43
*** silor has joined #openstack-swift18:44
claygacoles: top priority for me atm is to get the module level functions doing hash/suffix tracking moved to something more OO18:44
acolesclayg: so, i feel i'm lifting a bit of a stone with this pathc 16271718:45
acolespatch 16271718:45
patchbotacoles: https://review.openstack.org/#/c/162717/18:45
claygfucking colloquialisms18:46
acolesclayg: i like the idea that the diskfile impl knows to create a .durable or not18:46
* clayg goes to lookup "lifing a stone"18:46
acolesclayg: 'opening a can or worms'18:46
acoless/or/of/18:46
claygacoles: so how can I help - i agree it's a good idea - we're just trying to figure out the right way to do it?18:47
claygacoles: honestly thinking about what's going to make it and what's going to slip for the EC beta - the part I'm most concerned about is - we don't break anything that works today with replication18:47
acolesclayg: what i meant was, i now noticed that we do our hash_cleanup_listdir after writing the .data file, which with EC will do zero cleaning up because the latest durable isn't there yet18:48
claygif mutli-range GETs and COPYs don't work, if the reconstructor doesn't scale, these are all bad - but if we can iterate on the EC Policy sandboxed off where we won't hurt replicated storage policies - that's a win18:48
acolesclayg: so really the hash_cleanup_listdir should be delayed til durable is written18:48
claygacoles: hell yeah it should!18:48
claygacoles: I have another idea on two-phase PUT acctually :\18:49
acolesclayg: which is more making me feel like an EC-diskfile impl makes sense18:49
*** mmcardle has quit IRC18:49
claygYES +1 EC-diskfile makes sense18:49
claygwe didn't want to have to do it - but that was before we knew about fragment indexs and durable markers18:50
acolesclayg: ok we're just super-agreeing with each other. and yes, i get the need to land working stuff before worrying too much about 'elegance'18:50
claygacoles: no i'm in the other camp - i'd rather EC beta barely work and have a super solid framework to iterating on policies than have a crappy mess on master (and an EC beta that mostly sort of works if you're careful)18:51
acolesclayg: ok i misunderstood!18:51
* notmyname agrees with what clayg just said18:52
* clayg is shocked!18:52
claygthanks for having my back notmyname!18:52
clayglike really if people can benchmark PUT's and GET's and start trying the reconstructor at scale I think we have a good beta18:53
acolesnotmyname: clayg: and i didn't disagree, just misunderstood :D18:53
notmynameacoles: oh, I met Jim Meyer this week. told him that it's really great working with you and donagh18:53
claygI also think we're going to find issues no matter how hard we try - so we may as well get ourselves ready to fix those bugs without a bunch of refactoring and cleanup looming over our heads18:53
acolesnotmyname: thanks!18:54
claygacoles: so if you wanna play with the DiskFileHelpers and see what shakes out that's very helpful18:54
acolesclayg: ok, will do. Now, can i move to a really specific question?18:54
claygacoles: if you want to try and teach obj.server.ObjectController.get_diskfile how to route to policy specific DiskFiles - that'd be super helpful18:54
claygacoles: my favorite kind!18:55
acolesok here goes - agh, all my lights just went out18:55
claygoh no!18:55
*** fbo has joined #openstack-swift18:55
claygpeluse: oh... i guess you're traveling... anyway i'm going to rebase patch 159637 on acoles rebase of his patch that I stole from him18:56
patchbotclayg: https://review.openstack.org/#/c/159637/18:56
acolesclayg: ok, back on :) so DiskFileWriter does the data file rename and hash_cleanup_listdir in a separate thread18:56
jrichliacoles: I guess you have stayed late18:56
claygacoles: yeah it do18:56
claygjrichli: he likes me18:56
acolesclayg: not really understanding *why* that is, but should the durable file also be created in a separate thread?18:57
acolesclayg: like, same pattern for both18:57
acoles?18:57
acolesjrichli: clayg : i have to move every 20 mins after 6pm to trigger the light sensors18:58
claygacoles: i had no idea onus was a werd - sometimes i feel like I'm a total bafoon surrouned by the likes of you luminaries18:58
jrichliwe have auto lights going off at 6, but we just push a button to turn them back on for a long time18:58
acoles'move' as in, roll my chair across the corridor18:58
claygacoles: doesn't the fsync go down in there?  I think the idea is that blocks the hub18:58
claygos.listdir is blocking to if you're not keeping inodes in ram18:59
acolesclayg: yes, the fsync is done there18:59
claygacoles: ok - yeah so that's definately why18:59
acolesclayg: but i think we need to fsync the .durable too...18:59
claygacoles: i think we'd *want* to - yeah19:00
acolesso if i was to either (a) fsync the durable or (b) call hash_cleanup_listdir after writing durable, i should use a separate thread?19:01
claygacoles: yeah... idk - so doing two hash_cleanup_listdirs seems wasteful19:03
claygacoles: but we probably want to fsync the .data before we send the continue message19:03
claygfsyncs (and probably the listdir) should be in a thread to avoid blocking the reactor19:03
acolesclayg: no, i'm thinking an ec-diskfile would defer the cleanup until after the durable is written, so just one hash_cleanup still19:04
claygacoles: also just FYI - i tried sticking an exception error - maybe an oserror - something in those threads and the object server was not too happy about it - I didn't split off to look at it - but I'm pretty sure it's terrible19:04
acolesclayg: yeah, and we do want to fsync .data before the continue119:05
claygacoles: yeah i'm down with that - i'm just saying when you rename the - yeah you got it19:05
acolesclayg: hmm, i was also wondering what happened to OSErrors down in those threads :/19:05
claygacoles: i think the whole world breaks and everything goes to crap - basically what redbo and dfg are always complaning about19:06
claygwe'll probably finally get around to fixing it around the time they finally publish their go implementation :P19:07
acoleslol19:08
redboawww19:08
*** tgohad has joined #openstack-swift19:09
claygredbo: i'm just joshin' with ya19:12
acolesclayg: ok i think i need to go and play with the code some more - that was helpful, thanks!19:12
*** tsg_ has quit IRC19:13
claygacoles: i'm not helping - YOU'RE helping19:13
*** silor1 has joined #openstack-swift19:14
acolesclayg: oh ok, mutual hugs and kisses :P19:15
acoleswell, hugs.19:15
*** silor has quit IRC19:16
acolesclayg: notmyname : peluse : fyi i'll be quiet til Monday - taking time off for mother visiting19:16
clayglol19:17
claygok - have a nice weekend19:17
*** silor1 has quit IRC19:21
*** acoles is now known as acoles_away19:23
*** bkopilov has quit IRC19:27
*** thumpba_ has quit IRC19:30
notmynameredbo: dfg: please take a look at https://review.openstack.org/#/c/126923/19:30
*** bkopilov has joined #openstack-swift19:32
openstackgerritClay Gerrard proposed openstack/swift: Multiple fragment Archive Index Support  https://review.openstack.org/15963719:32
openstackgerritClay Gerrard proposed openstack/swift: Prefer policy to policy_index  https://review.openstack.org/16369019:32
*** nermik has quit IRC19:35
claygmattoliverau: i missed some of your comments on patch 163690 because acoles_away had done the rebase - but then it looks like he caught most of them - except for the import thing - which I would avoid because of the rebase pain19:36
patchbotclayg: https://review.openstack.org/#/c/163690/19:36
claygmattoliverau: maybe shortly after we cut Kilo we could clean everything up - but w/o a hacking rule it'll just get all mixed up again :P19:36
clayganyone think we should try H302!?19:39
*** openstackgerrit has quit IRC19:40
claygi wish abandonded changes didn't still show up in the Needed By dropdown - wtf good is abandonded if it doesn't get rid of them?!19:41
*** openstackgerrit has joined #openstack-swift19:41
redboI don't really have an opinion on that.  I don't think not fsyncing directories has caused us much grief, but from what I've been able to tell the performance drop isn't huge.19:44
*** os1 has joined #openstack-swift20:01
os1Hi20:01
os1From swift-recon output,20:02
os1What do the fields "Failed: ", "no_result", and "reported: " signify?20:02
claygos1: I'd guess it means timeout vs here's the data?20:12
claygos1: but if you're looking at it in context you may have a better guess atm20:13
clayg_ahale: you guys have some scripts built up around recon ya - do you use it a lot?  we mostly just gather up the raw output from recon dumps into collectd/graphite guages20:14
*** mahatic has quit IRC20:14
*** dencaval has quit IRC20:15
os1Where's the file where these are defined/explained ?20:19
claygos1: http://docs.openstack.org/developer/swift/admin_guide.html?highlight=telemetry#cluster-telemetry-and-monitoring mabye?20:21
*** nellysmitt has quit IRC20:22
_ahaleoh yeah i can never remember what that means, maybe the no result was for recon-cron stuff thats not run.. we mostly dump stuff to graphite too20:24
*** rdaly2 has quit IRC20:26
*** doxavore has joined #openstack-swift20:27
doxavorei've turned down the object-auditor to do at most 10 files/sec and 1MB/sec, but it still saturates my [new] 7200rpm SATAs (according to atop and iotop) - are there any other recommended settings, perhaps an ubuntu default, that people are known to change to help that along? :-/20:29
doxavoresaturate (v) - iotop and atop report the drive being audited as "100% busy"20:29
doxavoresimple dd tests on the drives do over 100MB/s20:30
sweeperhmmm20:31
doxavoresweeper: shhh. i don't look as brilliant when coworkers find i crowdsource all my problems :>20:32
claygzero_byte_files_per_second maybe?20:32
sweeperdon't have a whole lot of those...20:34
os1Do you know which code file the swift-recon outputs are defined?20:35
os1The docs are high level about it.20:35
claygsweeper: well it's an audit function that cleans up for rsync failures - it doesn't do the md5 checksumming - it just checks if the metadata content-length matches the os.stat size20:37
claygsweeper: doxavore: I just know it runs a seperate rate, it's weird you're getting 100% saturated - if you keep turning the knobs down will it *eventually* not say the drive is 100% busy?20:38
claygI think maybe there's a mis-reporting thing - like 100% doesn't mean what I'd think it means intuatively20:38
claygcan you vmstat the drives and see if the bytes in/out matches your settings?20:38
sweepermmm, that's a point20:39
claygI think *maybe* those settings can make it do less work per unit time - but never acctually doing *nothing*20:39
claygso it's 100% of the time doing *something* but it could acctually be doing *more* something in that unit time - maybe?  i'm splitballin' - it's a strange observation - i've not used those tools specificially.20:39
doxavoreatop has typically matched up with vmstat for me historically, but i will certainly take a look at that20:40
*** thumpba has joined #openstack-swift20:50
*** thumpba_ has joined #openstack-swift20:53
*** thumpba has quit IRC20:56
os1Asked this here earlier, but does anyone know which code file defines/explains what the fields "Failed: ", "no_result", and "reported: " outputted by swift-recon, mean?20:58
os1Thank you.20:58
claygswift.cli.recon20:59
*** cdelatte has quit IRC21:05
*** vinsh has joined #openstack-swift21:09
*** thumpba_ has quit IRC21:15
*** thumpba has joined #openstack-swift21:16
*** doxavore has quit IRC21:23
os1clayg : Could you clarify? I'm not sure where that is pointing to.21:29
*** tgohad has quit IRC21:29
os1clayg : I see. Thank you.21:34
*** cdelatte has joined #openstack-swift21:35
*** delattec has joined #openstack-swift21:35
*** NM has quit IRC21:39
srsakhamurinotmyname: I am looking for enabling oslo message bus notifications from swift, is using slogging framework a good approach to address this?21:45
notmynamesrsakhamuri: :-) I'm working on getting through my emails. can I answer that in a reply. I think I can do that later this afternoon21:46
srsakhamuriok cool, thanks was wondering whether my mail went to spam :-)21:46
*** jkugel has quit IRC21:47
notmynamesrsakhamuri: no. I've been traveling this week. back in the office today with about a 100 things going on at once :-)21:47
srsakhamuriI can understand, take your time. thanks again21:48
*** thumpba has quit IRC21:50
*** thumpba has joined #openstack-swift21:51
openstackgerritMerged openstack/swift: Bring EC API calls in sync with PyECLib 1.0  https://review.openstack.org/16353321:52
*** annegentle has quit IRC21:56
*** vinsh has quit IRC21:59
*** vinsh has joined #openstack-swift22:00
claygwhat does slogging (bad idea) have to do with oslo message bug notifications (terrible idea)?22:00
mattoliverauMorning22:00
claygwell... "bad" might be too strong of word - some people might see it as more attractive than rolling your own22:01
claygit'd be better if slogging was better - maybe send notmyname a big 'ol box of round tuits22:01
claygor just use slogging and keep making patches till it's good enough - it's no so terrible - i used to hack on slogging - i think we could have made it work :P22:01
claygglange: how far along did you guys get into the message thing before you decided to abandon slogging - I think I was on CBS at the time22:02
claygat least slogging is a good starting point22:03
claygany form of proxy log processing and and account data walking is better than olso messages22:03
*** vinsh has quit IRC22:04
notmynameslogging is great. recently dmorita send a bunch of patches and now it works with current swift22:06
notmyname;-)22:06
claygwell that's huge!22:06
clayg:P22:06
notmynameit ran RAX cloud files billing for years and still runs HP cloud billing. so that's something22:07
notmynamebut yeah. could be faster. could be refactored some. could be a lot of things22:08
claygnotmyname: it's nice that it's open source - it'd suck if there wasn't *something* out in the wild22:08
claygit always just ends up being more efficient to roll your own code because the *big* question is what you're going to do with the data22:09
notmynameif only the author would actually spend any time on it :-)22:09
claygslogging dumping to csv sorta punts on the issue - which was sort of a design decision IIRC22:09
notmynameya. which is where simple-billing at RAX came from (the part of slogging extracted to just get the billing metrics and do the calls to the internal billing system)22:10
*** tsg has joined #openstack-swift22:12
claygoh yeah!  I forgot about the SOAP stuff!22:13
torgomaticI step away for 10 minutes and we're talking about SOAP?22:16
torgomatic!?22:16
openstacktorgomatic: Error: "?" is not a valid command.22:16
torgomaticthat should either be a help command or it should just say "calm the heck down"22:16
jrichlilol22:22
*** jrichli has quit IRC22:24
*** logan2 has quit IRC22:26
mattoliveraulol22:28
torgomaticso, question for everyone: do you know of any application that uses multi-range GET requests to Swift?22:40
torgomaticthat is, GET requests that ask for multiple byte ranges, e.g. Range: bytes 1000-5000,178953-199185,...22:41
notmynametorgomatic: is there anything in a proxy log line which would make it a deterministic question?22:41
torgomaticnotmyname: not that I know of; single-range GET response and multi-range GET response are both 20622:41
*** thumpba has quit IRC22:51
*** thumpba has joined #openstack-swift22:51
*** annegentle has joined #openstack-swift22:56
*** Nadeem has quit IRC22:59
*** erlon has quit IRC23:01
*** annegentle has quit IRC23:02
openstackgerritRicardo Ferreira proposed openstack/swift: Created iterator for the metadata values, outputs the sysmeta and usermeta separately, each entry in a new line for grep parsing  https://review.openstack.org/16401923:25
*** os1 has left #openstack-swift23:39
*** david-lyle is now known as david-lyle_afk23:41
*** kei_yama has joined #openstack-swift23:43
*** thumpba has quit IRC23:52
*** thumpba has joined #openstack-swift23:52
*** reed has quit IRC23:55
*** gyee has joined #openstack-swift23:56
*** annegentle has joined #openstack-swift23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!