This is a long topic and you should probably read it all. You should also know that VFP 9 substantially enhances the
SET REFRESH command to, among other things, force VFP to check on disk for the latest information, and not rely on elapsed time as in prior version of VFP. Category 3 Star Topics.
How do you guarantee that when one user of a large, networked app updates a shared table, that another user sees that change as soon as it is made? This is not an abstract problem, but one which I've run into with my current client. The table in question holds semaphores used to prevent more than one person accessing an account at the same time, but -- despite all our efforts -- some users can still get in, since they don't see the changes of others until much later.
I've tried opening and closing the table, issuing
GOTO to force the pointer to move, and there is still a noticeable lag of up to a half-minute before some users see the change. The table in question is opened unbuffered, and the change is made via an
INSERT INTO or a
REPLACE (with an
RLOCK), followed by a
USE to close the table and a
FLUSH to really make sure that the changes aren't cached. But I think the Fox is too smart for its own good, and has outsmarted us. Anyone have suggestions?
The OS is NT4 SP4 (I believe), and the users are all running through a Citrix Terminal Server arrangement, so the actual hardware and OS being run on are tightly controlled (i.e., no weird desktop settings to affect app performance).
-- Ed Leafe
Thought I'd post some follow-up information. After trying all of the suggestions posted here, in all sorts of combinations, we finally settled on a methodology which uses
FLOCK() to force an update. My earlier worries about performance degradation seem to have been unfounded; the addition of the
FLOCKs has had only minimal effect on performance.
The general strategy is to minimize the duration of the lock. A typical section of code looks like:
Fixed the code to use
DATETIME() instead of
ltStart = DATETIME()
DO WHILE NOT FLOCK() AND (DATETIME() - ltStart < 2)
* Time out after 2 seconds
[access the data]
UNLOCKOf course, there is checking for failure to get the
FLOCK, etc. Since implementing this strategy in the app I originally started this topic about, as well as in another system subsequent to that, I have not seen the concurrency issues which plagued earlier versions.
Thanks to all who contributed their ideas! -- Ed Leafe
If you run many processes using this code at midnight, the system HANGS! This is because
SECONDS() will reset at midnight and
DO WHILE loops will not end after 24 hours. Please don't use this buggy code. Replace
DATETIME(). Andrus Moor
Hmmm. Maybe some of this explains why I'm having problems with a particular client getting duplicate PK values about once a week. The app was developed with VFE5.5 and running under VFP 6 runtime. Problem is that there is a candidate key that gets skipped when this happens. The candidate key is their internal "control" number so they get a little upset when it happens.
Here is the stored procedure that gets called. I'm using the SmartIncremental type:
* Function.......: VFENewKey
* Purpose........: Default Expression procedure for generating unique keys
* Parameters.....: Field, Character, Alias.FieldName for field to increment
* Return Type....: Unknown, depends on data type
* Author.........: Neon Software
* Copyright......: Copyright (c) 1996, Neon Software
#DEFINE VFESMART 4
#DEFINE VFEBASE10 5
#DEFINE VFEBASE62 6
lnSelect = SELECT()
luId = .NULL.
lnDBCXId = 0
IF TYPE("oMetaMgr") = "O" AND (NOT ISNULL(oMetaMgr))
lnDBCXId = oMetaMgr.dbgetdbckey(DBC(), "Field", tcField, .T.)
IF lnDBCXId = 0
lcKeyName = oMetaMgr.dbcxgetprop("FECKeyName" , lnDBCXId)
lnKeyType = oMetaMgr.dbcxgetprop("FENInitType", lnDBCXId)
lcType = oMetaMgr.dbcxgetprop("CBCType" , lnDBCXId)
* Assumes no decimal points in numeric fields
lnMaxLen = oMetaMgr.dbcxgetprop("CBNSize", lnDBCXId)
CASE lcType = "Y"
lnMaxLen = 15
CASE lcType = "B"
lnMaxLen = 16
CASE lcType = "I"
lnMaxLen = 10
lnOldReprocess = SET('REPROCESS')
*-- Lock until user presses Esc
* Changed to NOT allow ESC to interrupt lock attempt
SET REPROCESS TO -1
IF NOT USED("AppIds")
USE FESYS!AppIds IN 0
LOCATE FOR UPPER(ALLTRIM(cKeyName)) == UPPER(ALLTRIM(lcKeyName))
llError = .F.
IF NOT FOUND()
lcOnError = ON("ERROR")
ON ERROR llError = .T.
INSERT INTO AppIds (cKeyName, cValue) VALUES (UPPER(ALLTRIM(lcKeyName)), SPACE(lnMaxLen))
ON ERROR &lcOnError
IF NOT llError
DO WHILE NOT RLOCK()
CASE lnKeyType = VFESMART
luId = VFESmartIncrement(RIGHT(AppIds.cValue,lnMaxLen))
CASE lnKeyType = VFEBASE62
luId = IncrementBase62(RIGHT(AppIds.cValue,lnMaxLen))
CASE lnKeyType = VFEBASE10
luId = IncrementBase10(RIGHT(AppIds.cValue,lnMaxLen))
REPLACE AppIds.cValue WITH PADL(luId, LEN(AppIds.cValue))
SET REPROCESS TO lnOldReprocess
IF INLIST(lcType, "N", "F", "I", "B", "Y")
luId = VAL(luId)
Note that I've modified this a bit to NOT use REPROCESS AUTOMATIC but set it to -1 to try indefinitely and not let the user ESC (I was suspecting this was the cause). Anyway, even with my change they have had a unique key violation occur just today. Should I be using FLOCK(), FLUSH? Should I move the record pointer in AppIds before or after the UNLOCK? I have no idea at this point. They add hundreds of records a day to this table with multiple users, so percentage-wise, this isn't a high occurrence, but should not be happening nonetheless. -- Randy Jean
I'd run a COM component on one server on which each application registers itself. When a change is made that another client should see immediately, I'd call a method on the server that passes on this message to all relevant clients. I guess the Microsoft way would be to use the message queue server, or something like that. -- Christof Wollenhaupt
Now that sounds interesting. I'll have to investigate this a bit, as there are over 250 users running on a dozen Citrix servers, but at least this is an approach no one else had thought of before. -- Ed Leafe
I seem to remember that the only way to be sure you have up-to-date data is if you have a lock on a record or file. Could it be that the database is updated but the stale data is cached in the workstation? - Carl Karsten
I'm sure that's the problem; the question is how to guarantee that the version on the workstation is as current as possible. RLOCK() won't do it, because we'd have to first find the record to lock, and it isn't "visible" to the workstation at the time we SEEK or LOCATE. Since this is a semaphore table, if we find the record we know it is not available for editing, so there's no need to RLOCK it. We can't FLOCK(), as this would introduce some really nasty concurrency issues, and slow the system down tremendously. -- Ed Leafe
What about the
SET REFRESH command?
My problem was that records added to non-buffered (unbuffered) tables were not seen by other workstation which performs
select ... into table. They become visible only a number of other part readings of this table.
Howewer, after adding SET REFRESH TO 1,1 command changes are becoming visible immediately! -- Andrus Moor
Let's clear the air here.
SET REFRESH TO x,y controls TWO things: first parameter is for BROWSE windows, so leave it set to ZERO. What does it do? It refreshes BROWSE windows automatically at the interval set (it's in seconds). No one needs that in VFP. Oh, this setting does NOT affect the grid control.
The second parameter is extremely important and controls how old the local data buffer is allowed to be. The Default is 5 seconds, so if you've opened a table at 12:00:00 noon and issue This Form.Refresh() up to 5 seconds later, you will not read data from the network, but instead from your local in-memory buffered copy of the table. If you wait past that five seconds, you will get the latest data from disk. Note that this does not cause VFP to refresh every five seconds; it is an internal setting on the data you've cached so that it can be no more than five seconds old.
-- Chuck Urwiler
Okay, Chuck -- of course -- is right, but maybe not complete:
The second parameter remains as important as it already was, however, nitwitting (?) a little, makes me add the following :
When you (all) really need diving into things, it is extremely important that you realize/know, that only the data from the current block is updated ! Thus, skipping through a table only refreshes that block, which coincidentally (!!) is current at the 5 second (or whatever the second parameter is set to) interval. Hum ? Don't believe me ? install a sniffer. Ain't I right, I'm sorry, because then something else is influencing.
Note that indeed this is possible, because I've seen for sure that not all PC's on the same network and on the same data (table) behave the same. This, with respect to not updating a Browse-window ever (???) where the other PC's do, but ... when my not-understandable problem is there (ref. VFPCorruption ); anyhow, since VFP allows only for 1 as the smallest value, and skipping through a table will pass blocks for sure within this 1 second, we 'll have a problem guys. That is, if you REALLY want each record actual, you'll never never succeed, unless an =Inkey(1) is inserted ... (and SET REFRESH = n,1).
Confused ? I'll see you again ... Peter Stordiau
Here's what AlekseyT (MS VFP Team) had to say on SET REFRESH in UT message #815257 "Your application should initiate a record read operation in order to get refreshed data. VFP doesn't refresh all tables unconditionally. When it is about to read a record from a table, it checks whether the refresh interval has elapsed since the last time the table was refreshed. If refresh interval has elapsed, all internal VFP buffers for the table are discarded and VFP reads directly from disk or from OS cache. Simple GOTO RECNO() should refresh the record. ". I think this is relevant because it is current and assumed accurate.
It is also worth mentioning that I subsequently confirmed with Aleksey that he is referring only to VFP's "cache". The OS cache is a separate mechanism and, frankly, seems to be designed more for things like Word/Excel documents and their characteristic usage rather than a database application (doing frequent updates of only parts of files). -- Jim Nelson
I've seen it, Ed, but everytime I bring it up everyone calls me crazy. I think it's related to Win buffering. There was once a KB article about not using buffered databases in Windows but it went poof a few years ago (and no one else claims to have ever seen it). IMHO, Win tells Fox everything is cool when it really isn't yet, so updating is really slow. Solution? Dunno. Christof has an intriguing idea, though. -- John Koziol
It's definitely a problem with the interaction of VFP's buffering and Windows' buffering. All I can do from Fox is eliminate half of the lag. We've thought about creating a file-based semaphore system, since the OS would not allow identically named files to be created in the same directory, but the concern is that that would be way slow, so we're holding back on it unless it becomes super-critical. That's why I like Christof's idea - it might get around the lag while not being horribly slow. -- Ed Leafe
NT Server performs read-ahead, write-behind, and lock caching for optimization. Usually, there is no need to tune these settings unless the server is very busy, or something wrong with redirector. Try to search MSDN for oplock or usewritebehind.
Why don't you use traditional RLOCK?
We do. The problem is that when the lock is released, others accessing the same table don't see the changes immediately - they can SEEK or LOCATE FOR the new value and not find it. I've also added the NOOPTIMIZE clause to see if it was a Rushmore thing, but that didn't help either. -- Ed Leafe
"I've tried opening and closing the table, ..."
I don't think it's FoxPro. When you close a table, it still is kept in cache and when another user requests read, it should break oplock and cause update. Can you recreate the behavior if you run two instances on the server with minimal number of users?
We can't recreate the problem at will, but we can look at the Semaphore table at various times and find two users "locking" the same record. So it happens rarely, but often enough to be a concern. -- Ed Leafe
I seem to remember that the only way to be sure you have up-to-date data is if you have a lock on a record or file. Could it be that the database is updated but the stale data is cached in the workstation?
I think I can reproduce your problem. I have 3 programs that all run concurrently (even on the same machine, the results are the same.) LOCK0.PRG creates a table, inserts a record. LOCK1 changes its value every 3 seconds. LOCK2 and LOCK3 display the value, LOCK3.PRG gets a lock, LOCK2 does not. You will notice that the value displayed by LOCK2 does not change, but with LOCK3 it does.
LOCK4 is the same as LOCK3, with the UNLOCK done later. This demonstrates what could happen if you leave the record locked too long. LOCK1 spends more time trying to get a lock (10+ seconds), than waiting the 3 seconds.
I think you need FLOCK, which works the same as RLOCK in these examples.
GO.bat starts them, STOP.bat stops them.
They can all be found here: http://220.127.116.11/carl/temp/locks/
Not any more. /temp ??! what was I thinking? If anyone has a copy, it would be great if you could send it to me - I'll put it in a more permanent place.
Interestingly enough, I finally solved the age-old problem Ed described above. The concurrency issue dealt with "seeing" updated data immediately in a VFP table. No matter what I tried I still had a 2-3 second lag. That's killer when you have many concurrent users.
The solution is a 2-parter:
1. Use RLOCK() to gain immediate access to records. There is 0 lag for RLOCK() to be seen. When I want to access the next available record, I simply set REPROCESS to 1 so it only tries 1 time, then SCAN and check RLOCK() on each record.
2. Move the record pointer immediately (GOTO TOP, GOTO BOTTOM, GOTO lnRecno); otherwise, the change will never be seen as long as the user stays on that record. That's a fantastic feature, isn't it???
When I can finally get an RLOCK(), I have immediate exclusive access to that record. I then change the data and move the record pointer so that other users can see the data change in a few seconds.
This works like a charm! The only issue is that many users hammering a table with RLOCK() can cause slowdowns. If the users are doing something in between checks for new records, you should have no problem with this solution.
Chris- you might also consider using a SQL View whose TableUpdate() command will have the same effect as SKIP or GOTO RECNO(). Also makes it easier to upsize to a C/S database if you decide to - John Ryan
Below, you'll find a reply from me at another Forum to the question "how to prevent corruption at power loss ?".
This starts under the "========"; please bear in mind that the topic there was slightly different, but I think you will find some answers there anyhow.
First, some other stuff:
The most direct answer to your problem may be =Reccount(), because this actualizes the table the most directly. However, firstly the other PC's process must have "told" that the server is allowed to see the update (see text below).
Though this is stuff to break brains, in the for sure there is no problem, if you only write the app "decently".
But, for sure, note the following too;
When it happened, I don't know, but I'm pretty sure somewhere in time things changed according to all the rules which apply here, where our ERP app is stuffed with comments on this topic of about 9 years ago, then using FoxPro 2.5, which we still do today. And the comments nowadays don't fit anymore on certain sub-topics.
It is too much to describe here, but IMO this it due to client's parameters of nowadays, and which were not there before. So indeed, the re's above are probably very right for this matter. Mind you, a properly written app still won't have problems.
Now, please note the rather important fact that even Fox Plus was already capable of cacheing, where nowadays a (Win) client by itself does too; IMO, this is asking for trouble, and what we do is switch off all the OpportunisticLock-etc. parameters of the client. This, where other apps at the site allow this, but for us this is mostly the case, since ERP will be the major - and only heavy app, having priority above the rest (in speed, etc.). The switching off of these parameters is not the easiest way to do, because in the end it comes to the Registry where you have to find things, and -for instance- removing a (Novell / MS) client and installing another one, leaves things in the Registry. So, a rather hard job; but after some experience, you'll succeed in letting VFP (FoxPro) control its caching by itself. If you didn't create this situation, you'll never know what you see or may expect.
Something you possibly don't recognize is the fact that indexes are treated differently by VFP than tables. I mean, that were a table-record IS visible, an index-record not necessarily does. This is because VFP HAS to make clear to the other PC's that a record was appended to the table, because if not, your corruption is there.
Understanding this is rather easy, if you think of setting up a test-program that does runs at more PC's, all adding records, and where you will see that this never goes wrong. Thus, in any circumstance, the number of records is known to each PC, as long as it's asking for RECCOUNT(), which internally is done at the APPEND BLANK (or whatever like SQL Insert). But this doesn't account for Index-records, and why should it? Again, when the app is setup properly, the APPEND BLANK only leads to a blank Index-record (if it was already at the server), which can't be found by any normal Seek (and a blank key I don't find "normal").
A wrong application hoewever, should be a GOTO BOTTOM, and appending according to the data found in that record. Thus, suppose a sequence-number is in this last record, and the app says seqnr = seqnr + 1 and writes this back in a then appended record, for sure you find yourself in trouble. No matter how much you Lock or what.
In the end -and talking about a properly setup app- stuff like this should be in the (unique) key of the table, and things will go alright.
Please bear in mind that though Flush formally is there to flush amendments to the server, in fact this shouldn't be used, because it can never lead to any decency, and instead Unlock should be used, or Unlock All when you want to "flush" all. Also bear in mind that the Flush comes into action anyhow, whether you want it or not, after 5 minutes of no action, and Clear Events (note : I didn't test this for VFP, but suspect this, were in FoxPro-Dos for sure this is the case after leaving the Read). Anyhow, the FLUSH command is formally still there in VFP (5.0 anyways) but to my opinion should be removed (and for sure the 5-minute-thing !!).
Another thing : (see text below) :
When you setup 2 PCs where one appends records immediately followed by =RLOCK() and a REPLACE of data, and the other performing GOTO BOTTOMs and immediately Replaces something in the last record, you'll see that once in a while -and based on coincidence (think of a kind of time-slicing) the second PC indeed has data replaced in the appended record of the other BEFORE the first one has. This is really unavoidable, that is, to my opinion. But again, this is the situation noted before though little different (about the seqnr). Of course, an FLOCK() helps here (first PC), but anyone who thinks this brings the solution can wait for users to call that they have to wait so long for their record to be updated ...
Last small thing:
When the app heavily depends on
FLOCK()s already knowing that this will fail more than not, you may find yourselve on large response-times on the failing
Though I never found out what the real problem was here, I'm sure that a client can have parameter-settings which influences on this, and where in the bad situation you are not able to influence the time-trying of the Rlock(), leaving it with a 0,9 sec long tryal. Mind you that VFP allows for the appropriate parameters on this one, but sometimes (??) they don't work. Thus, when a locked record just is tested for whatever reason (for instance, to try if one is perfoming a certain process), this may take already a second. Now supposing that this happens 10 times, there you go. Of course if you really need the record this is no problem, and the user has to wait anyhow. But just for checking something, this is a waste of precious time.
Text from other forums (fora?):
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
You can Flush and whatever VFP tells about this stuff, it won't help you since the Win-machines are in between with their caching options. And, everything may even be more thick, looking at the (Novell ?) client also containing paramaters for caching, though everything comes together in the Registry.
In fact, for anyone who thinks that VFP's info should be believed, set up a loop on one PC adding records, and now look at another what happens. Anyone who does this, won't believe his eyes, and is sure something will go wrong, where for sure it won't.
You are dealing with two "variables" here, where one is the writing PC and its parameters on when the server gets the info, and the other PC having parameters on when to see this. Try reading the Help on
SET REFRESH, and you will be lost...
Of course, not anyone who knows already.
Now note that there are influences you won't read anywhere, but wich are thru anyhow;
If you setup the reading PC correctly, you may notice that whatever the writing PC does, the info to the reading PC comes to it per datablock.
Now I can write some pages to explain all this, but the most important thing is that this is an influence you can't control;
For the ones who already know (or think they do !), please think about the situation that the writing PC does not want to make the written records known to the server, that an infinite number of records is written, and that the write-cache of the PC is limited;
Don't think of any Set Transaction, and know try to explain yourself where the data will reside ...
You see, on the network, even if you don't want it ...
Now note that there are many more commands then VFP tells, allowing the reading PC to actualize its own cache, and, that the commands VFP tells us to do the job, don't always work at all.
When you set up a loop on the reading PC, try to play with commands as
SEEK, and in fact all DB-commands you can think of, and you'll see that with all of these commands you may get other results on how far the writing PC is.
No, please don't forget that the writing PC also has commands available to influence the possible view of the reading PC, such as (indeed) Flush, Unlock, and ... the datablock becoming full which is an influence, but which you can't influence.
On this subject, to my idea it's a miracle that all remains going well without corruption.
Bear in mind that the Reccount() is one of the best commands to actualize the cache of a reading PC, because it's always correct.
Note also that commands such as Append Blank implicitly performs a RLock(), which also actualizes the cache, and where you must try to see the influence of the explicit RLock(), which remains the record locked. Now don't get confused, where the Append Blank always leaves the record available to another PC, although it's immediately followed with an RLock(). What I say here, is that another PC is always able to fetch the justly appended record of the other PC, and which succeeds at block-boundaries, because when the block in the writing PC is full, it gets available to the other PC's, and the last record of the previous block still containing the blank record.
Though it may seems that I'm proving here that FoxPro is all or nothing, this -- for sure -- is not true, because with a normal properly setup application this gives no problems at all. Not even because of any coincidence-factor.
Now back to your question: you are dealing here with the ever actualizing of the Reccount() in the table-header, and which is necessary for the other PC's appending as well. Thus, where a writing PC does not make its new records known to the server, the Reccount() always does, implying (!!) blank records which are -a sort of- physically not there yet (think of a calculated offset).
So indeed, a
FLUSH will help, but a properly used Unlock does the same, but ... at record-level, where the Flush operates at "all tables" level, and therefore no being what you want anyhow.
Now bear in mind that you can
UNLOCK what you want, if the PC overrules this with cache-parameters, you ar still nowhere.
But note also this one:
If at the client, everything's working fine (including the program), you still may encounter this problem on a Novell server;
what it is, I don't know, but I'm 100 % sure that on a Novell server the following can happen :
The users work for several hours, and now the server goes down;
One of these users appended 20 tables in this time, and (only !) a few of these tables have the records missing after the server is up again.
Since our application performs an Unlock All at the beginning of each function, this is impossible, where other tables WERE updated;
one thing : other users may have enforced the updating of the indeed updated tables, whereas the one user has a general problem in its client (PS) that allows for no update at all anymore.
What I'm saying here, is that it's hard to prove that it's the server-OS causing the problem here, but one thing I know for 100 % sure: if the server stays normally up and running, there never is a problem of this kind. So now it's your (all) turn ...
because I can't explain this.
Also note that I'm talking explictly Novell here, since NT doesn't give this problem ever.
Now I also refer to my "Award" (ref. message of 24-03-2001), where I have this problem, and can't find the solution, but which solution fairly sure lies somewhere here ...
Start with applying Unlock's where you logically should, and normally you couldn't have any problems.
Where for me it's proven that you still have on Novell-OS, install a UPS, knowing that power-failure is normally the only reason Novell goes down. 90 % of these kind of problems are not there now anymore.
Where you still -- of course -- are confronted with server-down's, I can only give the hint to make the block-size as small as possible, which by the way is always to be preferred for DBF's (!); you'll lose less records now, because records are written at block-level.
Although the above is a rather brief description of my knowledge on this, I hope to have helped anyone who is confronted with the fuzzy world on this one.
And maybe some of you are encouraged to add some real important things here, so we all (and me) 'll learn someting again. Note that I have still my "Award"-problem ...
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Note on the latter : See other message of today from me.
New information about client2 failing to see change made by client 1!
I've recently had to do some lowlevel research into why changes to a record are not seen by another client already looking at the same table. Everytime this comes up and I have to do research on the web everyone seems so certain that their issue is because changes aren't getting pushed to the table. I learned just yesterday of a very real possibility in that the change gets pushed to the table, but because of the way VFP handles buffering, wether you have buffering turned on or not (we have it off), I noticed something using a simple test and Process Monitor from SysInternals... I noticed that the change did get pushed to disk, but when the secondary client references the same field it will read the header, get the updated date from there, and does nothing else except pull he value from the record block already stored in memory. There is no time context in the Updated (non Y2K date stored in the header) which it seems to be looking to determine if there was a change to the table. The reality is that you probably won't see the change until the next day when the date is different. I think this is a design flaw in VFP. This was all tested and verified in VFP8 which is what we still use for our production systems.
The simple test is open two VFP IDE's... Open the same table, let it land on the first record by default, do not allow any record pointer movement for both sessions. Change a field value in instance 1 and reference the field value i n instance 2. The only efficient way I can get instance 2 to see the change is to issue a CURSORSETPROP('BUFFERING',1) which will cause that instance to reread the current record. This is even though buffering is turned off, when you tell it to turn off again it will simply reread the current state of the record from disk
You can call Flush, Reccount all day long, the only things that I find to work are:
FLOCK() in client 2 not desirable
RLOCK() in client 2 not the best solution either
CURSORSETPROP('BUFFERING',1) in client 2, turn it off even though it's already off, it will reread the current record state from disk and will see the change. This seems to be the best non-obtrusive with least overhead approach that I can find. This will work even though client1 may have never left the record.
I also noted some discussion about a significant change to the SET REFRESH command in VFP9. When I performed the same test in VFP 9 with SET REFRESH TO 1,-1 so that it will always read from disk rather than cache...
It doesn't work either. In fact with the same test in this environment VFP9 doesn't even bother looking at the header or anything. There is absolutely no disk activity at all when client 2 references the same field that client 1 just changed. It will still pull the data from the record block already read into memory...
Refactored by: Gene Berger
Corrected typos and tightened up things a bit: Art Bergquist
Contributors: Ed Leafe Christof Wollenhaupt John Koziol Alex Dilman Carl Karsten Chris Probst Peter Stordiau
Category Big System