(20:34:17) Conversation with #sourceview at 2006-06-07 203417 on pbor
(20:34:17) You joined #sourceview
(20:34:17) irc.acc.umu.se sets mode: +nt
(20:53:48) barisione (~demian@81-208-36-94.ip.fastwebnet.it) joined #sourceview
(20:54:27) <pbor> ciao
(20:54:34) <barisione> ciao
(20:54:54) <pbor> intanto che muntyan arriva ti faccio il paste della mia nota tomboy
(20:55:00) <barisione> ok
(20:55:05) <pbor> NewEngineIssues
(20:55:05) <pbor> - async queue: offsets break. This is fixed in muntyan's tree, but a) the solution is not exactly trivial b) the solution is potentially expensive if the queue grows large (think of search&replace)
(20:55:05) <pbor> - highlighting breaks: we thought this was due to the queue issue, but there are still problems which we cannot pinpoint to an 'obvious' cause. In particular see the "debug thing"
(20:55:05) <pbor> - performance: even with the queue fixed and despite there are some good euristics to speed up the 'interactive' case, we still can come up with cases that fool the euristic: if 0.
(20:55:10) nud (~sf@d83-182-30-136.cust.tele2.be) joined #sourceview
(20:55:57) <barisione> I don't understand the second point
(20:56:19) <pbor> barisione: the 'hl breaks' point?
(20:56:27) <barisione> yes
(20:56:33) <pbor> barisione: do you have a checkout of muntyan repo?
(20:57:01) <barisione> i'm downloading it now
(20:57:07) <pbor> ok
(20:57:15) muntyan (~muntyan@pool-71-113-224-59.herntx.dsl-w.verizon.net) joined #sourceview
(20:57:17) <muntyan> hi there
(20:57:36) <pbor> basically there is a (or maybe more) cases where the hl is not properly updated
(20:58:08) <pbor> those cases were crashing before the queue fixes, so they were 'hidden'
(20:58:13) <barisione> hi muntyan
(20:58:32) <pbor> muntyan: I pasted barisione the tomboy note I shown you before
(20:58:40) <muntyan> good
(20:58:59) <muntyan> so, who's talking?
(20:59:20) <muntyan> pbor: you rule the meeting
(20:59:25) <pbor> ok
(20:59:45) <pbor> I'd say point 1) the queue thingie is mostly a statement
(21:00:05) <pbor> so there is not much to discuss about it (at least at the begining)
(21:00:17) <pbor> there was an issue-
(21:00:22) <pbor> there is a solution
(21:00:46) <pbor> the solution is not very nice so we need to evaluate its cost
(21:01:03) <muntyan> err, i have a correction
(21:01:11) <pbor> ok, shoot
(21:01:38) <muntyan> we do not know if it's a solution; and we do not know what exactly to do with the queue, since sorting doesn't seem to be enough
(21:01:56) <muntyan> barisione: what do you think about this, will it work?
(21:02:20) <pbor> muntyan: yeah, that was what I was coming to...
(21:02:43) <muntyan> well, you started far away from it then :)
(21:02:50) <pbor> I was trying to introduce stuff: the summary is
(21:02:56) <pbor> we have a bunch of issues
(21:03:06) <barisione> (I was reading you last commits)
(21:03:25) <pbor> they are not normal bugs, but are fairly deep in how things works apparently
(21:03:34) <pbor> so my idea was:
(21:03:49) <pbor> - list the issues (at least the ones that we know)
(21:04:05) <pbor> - list the possible approaches to correct them
(21:04:23) <pbor> - evaluate if the result is not an ugly hack
(21:05:06) <pbor> obviosly if we can't fullfill the second, we need more drastic decisions and there is no point discussing the third :)
(21:05:55) <pbor> the issues we know are the ones I pasted above
(21:06:01) *muntyan hopes the queue will save us
(21:06:31) <barisione> muntyan: in you last email you wrote:
(21:06:31) <barisione> The obvious one is to have timer and make update_syntax have
(21:06:31) <barisione> an extra argument - the time it's allowed to spend. It doesn't look
(21:06:31) <barisione> pretty though.
(21:06:42) <barisione> how do you plan to do it?
(21:07:52) <muntyan> well, the idle_worker creates a timer, and calls update_syntax(..., how_much_time_left)
(21:08:00) <muntyan> in a loop
(21:08:15) <barisione> how big should be the batch you read?
(21:08:18) <muntyan> update_syntax in turn processes batches of fixed size until it runs out of time
(21:08:27) <muntyan> i think 4k chars or so
(21:08:33) <barisione> you don't know how much text you will need
(21:08:34) <muntyan> the initial batch size looks good
(21:08:42) <muntyan> but i guess it needs experimenting
(21:09:06) <muntyan> right, i don't know
(21:09:12) <muntyan> but it's not a problem
(21:09:23) <barisione> i tested gtktextbuffer, reading a big batch is two time faster than reading it line by line
(21:09:41) <barisione> it's for this reason i wrote the ugly LineReader stuff
(21:09:59) <muntyan> i read line by line in my thing, and the most time is spent in regex searching and redrawing
(21:10:20) <muntyan> so i don't think line vs big batch will be an issue
(21:10:24) <muntyan> plus, why line?
(21:10:39) <muntyan> many lines at once, how many will fit into a batch
(21:11:31) <muntyan> anyway, this is *not* the most important issue, imo
(21:11:32) <barisione> I tested it line by line, it's the opposite case of reading a big batch
(21:11:38) <barisione> it was just testing
(21:11:54) <muntyan> it doesn't affect how we deal with modifications; and modifications is the thing that needs to be fixed
(21:12:27) <muntyan> i mean, once we know how to make it *work*, we can think how to make it work fast and nice
(21:13:09) *muntyan is not good in those idle things and batches, the stuff about loops and timers is just some vague ideas
(21:13:21) <barisione> however i like you idea on how to modify the engine
(21:13:53) <muntyan> barisione: tell more :)
(21:14:16) <muntyan> barisione: you know the code, i know only some tiny bits
(21:15:17) <barisione> I don't know how to handle modifications that don't fall completely in a batch
(21:15:57) <muntyan> looks like splitting them should work fine
(21:16:37) <muntyan> say, we process now chars from 0 to 1000; if there are twenty chars deleted from 990 to 1010, it's okay to process first deletion from 990 to 1000, and then from 1000 to 1010
(21:16:52) <muntyan> since those chars after 1000 won't fall into the batch, and analysis won't touch them
(21:17:35) <barisione> yes it should work
(21:17:36) <muntyan> (we do need to avoid analyzing-whole-remainder)
(21:18:42) <muntyan> what i worry about is that 'CHANGED' modification. i looked at the code, and apparently it should just work (with corrections in the places that modify offsets in the tree). but i don't know if it works. do you think it will work?
(21:19:39) <barisione> i'm trying this approach on paper :)
(21:19:49) <muntyan> good :)
(21:19:54) <muntyan> you know what to draw :D
(21:24:31) <muntyan> by the way, LineReader is an auxiliary thing to avoid slow reading from buffer, and it can't be removed if we don't care about the performance (i don't want to remove it, i want to understand what it is). correct?
(21:24:45) <muntyan> err, it can be removed
(21:24:58) <barisione> yes
(21:25:22) <muntyan> then we can safely read big batches regardless of how much we analyze
(21:25:49) <muntyan> so the reading lines vs reading reading a batch won't be an issue
(21:26:02) <barisione> it's here because reading line by line is slow but reading a big batch is often not needed if the user just inserted a char
(21:26:11) <barisione> if you read big batches it works
(21:26:19) <barisione> without any problem
(21:26:29) <muntyan> well, it does read big batches now, and it doesn't seem to hurt; so reading big batches is good
(21:26:42) <muntyan> (maybe not huge batches, of course)
(21:27:46) <muntyan> it's better to read big chunk since after inserting a char we potentially modify big region. so if it's not expensive to always read big chunk of text, we should do it. and it doesn't seem to be expensive
(21:28:23) <pbor> muntyan: well, but you analyze the chunk line by line anyway... no?
(21:28:31) <barisione> i tried it and the engine was slower but gtktextbuffer seems faster than one year ago
(21:28:32) <muntyan> pbor: yes
(21:29:01) <muntyan> barisione: no no, i have gtk-2.6 as minimal requrement :)
(21:29:16) <barisione> are you using 2.6?
(21:29:17) <muntyan> pbor: my department got gtk-2.6! no more gtk-2.2!
(21:29:22) <muntyan> barisione: yes
(21:31:03) <barisione> i tried to read 10K of normal C code and reading it line by line was really slower than reading it in a single batch
(21:31:32) <pbor> mmm
(21:31:35) <muntyan> so let LineReader do its job!
(21:32:36) <pbor> which is a bit weird since TextBTree stores segments line by line, so reading a whole block should just read line by line and concat...
(21:33:29) <muntyan> maybe offsets are the problem?
(21:33:57) <muntyan> hm, if there is an iter, then forward_line should work fine
(21:34:12) <muntyan> well, maybe it was a textbuffer bug
(21:34:17) <muntyan> it's full of bugs, you know ;)
(21:34:25) *pbor knows
(21:34:32) <muntyan> anyway, it's not the most important thing!
(21:34:37) <pbor> indeed
(21:34:56) <pbor> I am most concerned about the "debug thing"
(21:35:04) <muntyan> what I need now is to know whether my approach is feasible so that I try to do that stuff. if not, i need to know what to do :)
(21:35:41) <muntyan> i hope that debug thing is caused by modification which falls into the batch but is not processed in it
(21:35:53) <muntyan> (it is the case with debug thing)
(21:36:11) <muntyan> barisione: what does your paper say?
(21:37:07) <barisione> your idea should work, CHANGED should work
(21:37:32) <barisione> but how to you plan to use the old tree?
(21:37:59) <muntyan> heh, this is what i was asking about :)
(21:38:13) <muntyan> i have no idea how to use it, i hoped that the code will just do it right
(21:39:07) <muntyan> it can merge new nodes and old nodes now for deletions and insertions, right? i hope that it will just do it for 'changed'
(21:39:57) <pbor> well, as far as I can see the tree doesn't care about insert and delete
(21:40:05) <muntyan> hm, actually, this can be easily tested by manual injecting CHANGED modification into the queue and seeing what happens
(21:40:05) <pbor> so it should not care about changed
(21:40:13) <muntyan> yeah, it only adjusts offsets accordingly
(21:40:35) <pbor> yep
(21:40:35) <barisione> yes for changed it should work because, as pbor said, it doesn't care about insert or delete
(21:40:54) <muntyan> barisione: could you tell in couple of words how tags are applied and removed in the reanalyzed regions?
(21:41:14) <barisione> but if the tree cannot be used do you plan to discard it as the current engine does?
(21:41:29) <muntyan> no, i don't want to discard everything
(21:41:45) <muntyan> i think it should keep trying till the end
(21:43:30) <barisione> the code to apply tags is mostly copied from the old engine
(21:43:45) <pbor> as far as I understood from the code, the reasoning is that if doesn't merge "soon" it will prolly not merge at all
(21:44:00) <barisione> pbor: yep
(21:44:06) <muntyan> err, is it really the case?
(21:44:20) <barisione> I copied from other engines (colorer?)
(21:44:55) <muntyan> well, i personally don't see a reason to give up after one batch
(21:44:56) <barisione> after something like 100 lines of code is really really difficult to reuse the old contexts
(21:45:27) <muntyan> hm
(21:45:51) <pbor> barisione: does colorer use a full tree like the new-engine?
(21:45:52) <muntyan> the tree is mixed with the text
(21:46:29) <muntyan> i like more an approach of having tree and the segments separately
(21:46:35) <muntyan> then it is easy to reuse old stuff
(21:46:59) <muntyan> since two segments are 'the same' if and only if they point to the same node
(21:47:07) <muntyan> but it would require lot of changes
(21:47:31) <barisione> I don't remember exactly how colorer works
(21:52:17) <pbor> barisione: any opinion on the debug thing miscoloring? do you think it's related to the queue?
(21:53:34) <barisione> i can't test it now as I'm using windows
(21:53:50) <pbor> okay
(21:54:12) <barisione> and i cannot reboot as booting in windows does not always work on my old laptop
(21:54:23) <pbor> no prob
(21:54:59) <pbor> the third issue in the list has to do with batch estimation...
(21:55:06) <barisione> and sadly i need windows until i finish a stupid vnc clone written in C# for university  :(
(21:55:35) <pbor> (or least that was our guess)
(21:55:44) <pbor> barisione: server or client?
(21:56:03) *pbor would love a decent vnc client, even on mono :)
(21:56:27) <barisione> server and client, it's similar to vnc but it doesn't use the vnc protocol. it's not really usable
(21:56:30) <muntyan> so, what about the simplest thing now: adding CHANGED, sorting the queue, getting the modifications in a batch correct, and leaving the rest as it is
(21:56:46) <barisione> vnc works surely better
(21:56:54) <muntyan> then we can see what happens, if it works we can think of batch size and stuff
(21:57:15) <barisione> muntyan: I agree with you
(21:57:17) <pbor> muntyan: sounds like a plan
(21:57:29) <muntyan> good
(21:57:47) <muntyan> then i'm leaving. real life calling :)
(21:57:54) <pbor> ok
(21:57:54) <muntyan> thanks guys, see you, etc.
(21:57:55) <barisione> bye
(21:58:13) <pbor> thanks for your help marco
(21:58:18) <barisione> pbor: I need to ask you something about ErrRegex
(21:58:23) <barisione> EggRegex
(21:58:35) <pbor> sure (though I know little about it)
(21:59:37) <muntyan> oh, i forgot about that! i have a question about regex too
(21:59:57) pbor set topic on #sourceview: regex!
(22:00:03) <barisione> muntyan: ask you question
(22:00:12) <muntyan> barisione: remember there was a disucssion in bugzilla about offsets vs indices; and you with mclasen seem to have agreed that it should be changed to use offsets
(22:00:27) <muntyan> barisione: so, did you do it in gtksourceview's eggerex?
(22:00:35) <barisione> it's what I was going to ask :)
(22:00:57) <barisione> mclasen, paolo and I preferred offsets
(22:01:05) <muntyan> don't change it! there are people (me) who use eggregex a lot, as it is
(22:01:07) <muntyan> :)
(22:01:29) <barisione> but it has several problems as pcre always uses indices
(22:01:38) <barisione> are you using offsets or indices?
(22:01:46) <muntyan> indices are better because it's easier to work with the subject string when you have indices. you don't have to use g_utf8_stuff
(22:01:52) <muntyan> i am using indices
(22:02:23) <muntyan> if one works with "words" and "chars", then offsets are better. but who in C works with "words" and "chars"?
(22:02:46) <barisione> the current version uses offsets
(22:02:59) <muntyan> um, then i have old version?
(22:03:14) <barisione> the version in gtksourceview
(22:03:24) <muntyan> specifically, egg_regex_fetch_pos()
(22:03:33) <muntyan> does it returns char offsets or byte indices?
(22:03:46) <muntyan> (my copy returns bytes)
(22:04:29) <pbor> is the api big? to me it would sound sane have both...
(22:04:55) <muntyan> hm, offsets are good for textbuffer
(22:05:03) <pbor> yep
(22:05:04) <muntyan> since you can't work with bytes in a textbuffer
(22:05:15) <pbor> my point exactly
(22:05:15) <barisione> the version in gtksourceview uses chars
(22:05:29) <barisione> i converted both the engine and eggregex to that
(22:05:50) <muntyan> api is quite big, like almost every function is affected :)
(22:05:56) <barisione> the old engine was really ugly because it kept in memory both indices and offsets
(22:06:04) <pbor> so it means that muntyan copied eggregex from sourceview... so it's using it patched
(22:06:18) <pbor> s/it/he
(22:06:34) <muntyan> what do you mean?
(22:06:51) <pbor> <muntyan> i am using indices
(22:07:03) <muntyan> no, i am using non-patched version :)
(22:07:03) <pbor> gah
(22:07:07) <pbor> sorry
(22:07:15) <muntyan> i have eggregex copied long time ago
(22:07:20) <barisione> contextengine.c uses only offsets, converting it back to indices (if we decide to do so) should not be difficult
(22:07:20) <muntyan> before it was converted to offsets
(22:07:29) <pbor> yes
(22:07:40) <muntyan> well, converting my code shouldn't be difficult either :)
(22:07:47) <muntyan> whatever is better will be good
(22:07:53) <barisione> in the cvs on sourceforge you can find the conversion from indices to offsets
(22:08:00) <muntyan> and perhaps offsets are really better for textbuffer
(22:08:19) <muntyan> barisione: you mean eggregex changes?
(22:08:22) <barisione> yes they are better but they could be terribly slow!!!
(22:08:31) <barisione> no changes to the engine
(22:08:45) <muntyan> barisione: i am not using gtksourceview :)
(22:08:54) <muntyan> are they slow or they could be slow?
(22:09:40) <muntyan> hm, wait
(22:09:44) <barisione> my version of eggregex needs to do some conversions from offsets to indices but it the text you are analyzing is big than this is slow!
(22:09:46) <muntyan> if engine uses LineReader
(22:09:55) <muntyan> then it uses char*, and it doesn't care about textbuffer
(22:10:03) <muntyan> offsets are needed only for tags
(22:10:15) <muntyan> ah, i got why you kept both offsets and indices
(22:10:28) <muntyan> hm, situation
(22:11:41) <muntyan> okay guys, now really have to go
(22:11:44) <muntyan> see you
(22:11:50) <barisione> bye muntyan
(22:11:57) <barisione> i will wrote a mail on this
(22:14:51) <pbor> ok
(22:14:55) <pbor> I have to go too
(22:14:57) <pbor> bye
(22:15:16) <pbor> I'll pass the logs to paolo if you are ok
(22:16:23) <barisione> ok
(22:16:26) <barisione> bye
(22:16:52) You are now known as pbor|out
(22:27:06) barisione (~demian@81-208-36-94.ip.fastwebnet.it) left #sourceview
(00:10:03) nud (~sf@d83-182-30-136.cust.tele2.be) left #sourceview
(01:23:25) muntyan (~muntyan@pool-71-113-224-59.herntx.dsl-w.verizon.net) left #sourceview