[Synckolab] still has speed problem
niko.berger at corinis.com
Fri Mar 7 03:50:25 PST 2008
Christoph Amthor wrote:
> Hi Niko,
> I just went through the description. Just for interest, probably not
> important: (sorry if it's already there and I overread it!)
> a) Under 6. you write: if we did not find an ENTRY, save CUR in the adress
> book/calendar and continue.
> You probably check before if CUR is in the LOCALDB, to make sure that ENTRY
> was not deleted before?
yea, the check is there (see the last point of 7.)
> b) 9.: if ENTRY is in LOCALDB: delete it --- it will also be deleted in
> LOCALDB, I assume
> Now, is there any different way for the first run? If the difference between
> deletion on one side and new entry on the other side can only seen by the
> Localdb, then you need something different for the 1st run. Since you might
> have more than one client connected to the server, just copying all from
> client or server to the LOCALDB will not help, I think.
On a first run basis, I dont delete anything (since there is no
localdb). so it will just add everything either locally or on the
server. If it finds problems it will ask, and form then on there exists
a localdb to check against.
> This is probably in general one of the core tasks how to see if an entry is
> new or has been deleted before..
> Another problem I'm not clear about:
> If you have two clients connected to one server, and imagine you have one
> entry which is synchronized, so that it is on all three of them. Now you
> delete the entry on one of the clients. With the next sync it also gets
> deleted on the server. But now you sync the server with the other client and
> this client still has the entry and also has it in its LOCALDB, so it thinks
> that it is new (created locally after the previous sync) and puts it on the
> server. And then the entry will return also to the first client, where it
> was initially deleted.
No. If you have two clients (A,B) then you have following setup (for an
now you delete the entry on A and sync:
Aab(missing) -> Alocal=Server => delete Alocal and Server
now you sync on B:
Server(missing) -> Bab=Blocal => delete Bab and Blocal (see 9. first case)
> This would mean, that you cannot delete an entry with two clients, because
> it always gets erroneously "refreshed" from the other side.
> I thought maybe it would be good, if all (clients and sever) have a list of
> entries where they are marked as new/changed or as deleted, each entry in
> the list with a time stamp.
Actually there is a list like that, and it also includes a timestamp
(the XXX.ab.hdb) in there i save the email header information like
timestamp, message size and uid. thus whe i sync i jsut compare the
entry in this file with the server header reported by thunderbird and if
they match, i just use the locally saved file (this also makes sure
Server=LocalDB) and speed up syncing A LOT
> With synchronization, the lists on both sides
> are compared and the newest entry will be valid. Lists are updated on both
> sides accordingly. It's important also to keep a record of deletion for some
> time in case there's a client that synchronizes very seldomly. Of course,
> the localdb pobably does something like that, but we need to have it also on
> the sever and have to keep track of entries deleted long ago.
Hmm... this idea sure is interesting. Lets play around with it:
You would need another folder specifically for transmitting sync data.
In this data file you need one big file containing a hash of all local
entries. When you sync you just need to download this one file with the
info and compare the local entries with this file.
Problem 1: calculating a hash already takes most of the time anyways. As
long as there is no valid timestamp in the contact entries there is
nothing i can do except actually go throug ALL the data - comparing it
with the db every time - same as its already done.
Problem 2: the parsing and downloading of this file already consumes
quite a lot of time
Problem 3: this will ONLY work if you use the same synckolab version and
no other client (like kontact or outlook or web-based clients)
Especially Problem 3 and 1 make this idea unpractical. In order to work
around this problem I suggest:
* create a seperate sync profile for contacts. Put them on autosync and
write in there something like 180 minutes (or even more) - background
sync. Then it wont sync so often. thus you shouldnt notice it as much (I
already promised some that i will include an option that allows to
adjust the delay between each message process in order to give the cpu
more breath so you dont notice the sync as much)
* for calendar sync you can keep a short sync time (seperate profile).
Since you can specify an area in time it synckolab should touch. there
should never be as many entries there (say like half a year, or if you
are a busy man quarter a year). This should then go MUCH faster.
I am afraid I included all the optimizations I can think of. Maybe if
speed up more. Also having a valid timestamp on contacts to find out the
ones that definitely have NOT changed would help a lot. Lets see what
the future brings.
>> there is a description about the hwole sync process on
>> http://www.gargan.org/extensions/synckolab.html (around the middle:
>> How does Sync Kolab sync your entries?)
Thanks four your suggestions and ideas. If you find anymore, keep me
posted - I may have overlooked something :)
More information about the Synckolab