Server player numbers access

[Makzimia]

Greetings all,

I have reached a conclusion that decisions have to be made again in light of the much appreciated popularity of Fredian.

While it is nice to see such large numbers on, along with that, does come lag for some, and in very large numbers for a NWN server with a very large world, comes even more it would seem. No matter what bandwidth I throw at this, it will make no difference, no matter what hardware I throw at, it also won't.

So, the question is, what do I do next? the first thing I have done is lowered acceptance numbers down to 55, this seems to be the saturation number before lag sets in for everyone, we reached 59 at one point today I am told, and lag was on for one and all.

I cannot control the speeds of incoming players, but I can control how many are on the server, and whether or not we look at other ways to split the numbers up to more manageable proportions. At 57 players I saw CPU use of between 75% - 98% with times of plateau at 98% this was when lag would have been apparent I would say. Number of clients being large with averaged out large rates of ping up the lag, is what I see happening.

So, where do we go from here? I am open to suggestions, but I am loathe to remove our gamespy listing as I did last time, as natural attrition causes no fresh blood to be available. The second server is lower spec but splitting numbers across the two with less areas on each portalled across MAY be an option IF a way can be found to also communicate across the two servers. Otherwise I do not favour that way.

I am also faced with losing my test bed that I can throw up and down then, and I do not feel inclined to invade anymore of my personal machines for the sake of server dev use & access.

I am sure this will be a hot topic, and I am wanting a valid set of views and no bias against ANY groups of players in this.

I leave it to you, final decision as always will be mine, but let's open dialogue on it at least.

Makz.

[Silentus]

I know Makz expressed reservations with this, but I am 100% for removal from GameSpy for the time being. I agree that we want a constant infusion of fresh blood - but I believe that we will get some of that directly from your high rankings in NWVault (however messed up some may be). Further, we can always repost it if things begin to stagnate.

As far as the server split - I agree that if we cant send tells from one to the other (which I dont think is possible) I would also hate to see it. IF that route is opted for however, I would propose a STRONG push that our community (yes that means all you) foot the bill for a new server via donations. It is unreasonable to exepect Makz to eat that cost, and it is equally unreasonable (considering his care of our world) to simply to do away with it. As I said though - I would not like to see this unless cross server communications are possible.

[Jorthain]

If it comes down to a second server. I am sure that Parwith & I could donate some hardware or money as the case may be. I say remove it from Gamespy. Parwith & I found the server via the vault anyways.

[ride]

I know!! we can go hunt down all the 56k.... uh... nevermind
I'm kidding *really*


Makz, just drop the player #'s for now, and leave it at that... that will tell us if that is what was causing the lag or at least contributing to the lag, and we can live with a couple fewer player slots....
other than *that* do nothing else for now.... change _1_ thing at a time and then see what happens, tis the nature of troubleshooting.

[Makzimia]

Trying to preach to the converted ride? I am an old hand, I just like to make all things possible plain for all. I think I am covering my bases.

Makz.

[addicted2rpg]

Well, time for all hell to break loose.

I proposed at one time (it was a bit premature at the time) about finishing up Fredian in a whole new way.

The idea is to migrate players out of the island into a portal server that would contain new areas. It could be one of the small islands surrounding Fredian, an extension of Fredian, or as I suggested: Fredian's piece of the Underdark. The drow have always been a big Fredian theme, along with the warped, from the very beginning. Although there has been all kinds of crazy stuff about warped kingdoms, the true story as I have heard it was they were just regular folks who got warped by the high plane activity around Fredian. Who knows? Maybe Sir Ganfry was a mass murderer in another life and got warped into the good standing man he is (with all the uber powers along with it too).

The problem as Makz has told me is that the module is getting big and NWN isn't handling it well. A new module covering "the second half of Fredian" would solve our expansion problem and allow a lot of growth. We can also have a portion of players who at any given time may be doing something on the second half (maybe even just to avoid the lag, har!) and thus reducing lag for us. I'm sure all kinds of things like crafting etc.. will be made available for it.

The only concern is code-synchronization. Any script updates in one has to be done in the other one. Database stuff shouldn't be a problem but I might have to make sure to remind Makz to have the portal machine get access to the database machine. For load balancing, ideally three machines (one doing the database, the other two accessing it from their respective modules). But I would assume a new mod would be less crowded than the old one, so maybe it should host the data?

There is the alternative of running dual copies of Fredian, with one machine's NWNX setup for accessing the DB on the original machine. However, the growth benefits wouldn't be there, and the possibility of two events happening in West Styne at the same time become possible. From an RP point of view, its weird referring to "the other Fredian" DM events would have to specify which server, etc... but if we used a new mod, we can simply reference the area like always, whichever mod that area falls in is just a matter of whether or not the player knows about it from their explorations. They won't have to worry about being on "The right Fredian".

This has been done before.

In the action section, take a look at the Aventia servers. Aventia, like Fredian, grew to epic proportions and they sundered the mod by area and created *5* servers that now hosts the world of Aventia. Naria, Dusk, etc....

I don't think script/blueprint synchronization will be that bad, but Makz would be the one to live with it and he is already stressed out and I don't want to make his workload worse. There is also a little matter of asynchronous scripts. For example, the jail system would have to sever portal the user if they got jailed on the server where the jail is not at. GetModuleName() makes this easy. A script can ALWAYS know which mod it is on. The book of kehaar would probably have to be worked out as well. Anything involving movement or jumping players to a location would have to be updated for a portal system if we were to do a new mod. Its quite an undertaking, but I'm willing to update my stuff for it.

But before we get into "making the underdark," as opposed to "the third isle offshore", or "the rest of Clevian", lets just discuss a new content vs. duplicating content for portaling.

I'll summarize the needs:
1. script synchronization (import for one, import for both)
2. asynchronous scripts need to detect which mod they are on (kehaar, jail system, slave, anything that warps a player. I can take care of my stuff; not sure about kehaar. I could look at it later and make an assessment of how hard it would be to convert....)
3. database must only be running on one machine, the other must access that machine through Makz's LAN for its database calls. Myself and Makz's wife both know enough to set this kind of thing up.

I actually do all my development in new mods. Although I have a stripped down version of Fredian, I export the resources I need to an .ERF, make a new module, and import them. Then I set down and do development. As a result, my work / update .ERF's are very independent and would synchronize.

My only other concern is clobbering. I'm not the only one submitted scripts anymore. If I update these scripts:
A
B (dependent on A)
C (dependent on A)

and someone else updates:
A
D
E

and their update is loaded last, it could clobber the changes I made in 'A' and cause B and C to fail. At work, we deal with this by "check out". Its like a book in the library. We "check out" the file and no one else will modify it until we check it back in. Because we are working on completely different systems right now it is not a big deal, but this is a heads up for multiple developers. I think I am going to start tracking the files I am using in the development thread so my comrades have a heads up...

Of course, clobbering is a concern we already have now. It isn't caused by multiple mods.

[Phoenix]

What if the server was split differently than you currently are thinking (take the world and cut it in 1/2 or make a copy). What if all crafting rooms loaded to the secondary server. So places like Shard's and the various crafting halls were on the second server. That would split the server based on crafting/non-crafting activities, so all RP happens on one 'fredian' isle.

It would make crafting a hair harder, since you would have to load a new server in between getting ingredients (pelts, ore, whatever) and crafting, but isn't this usually done in batch by crafters? IE Collect ingredients until inventory is full or your walking at the speed of a snail in a salt maze.

I know crafting is a lot of overhead on the server, so this might help a lot, even if only a smaller percentage of the people connected were crafting. This would also move chests to the other server, since most of them are located in said locations.

We could have small crafting rooms for single items, maybe something similar to the old forge/anvil in W. Styne, but in another location. It could even have a pop-up notice on enter saying 'this area is for crafting single items only, not training' or something, so it's a server rule, and clear. Many people crafted in W. Styne because it was more of a law than a server rule (thus an alignment question for some).

[Makzimia]

Thanks for the comments so far on this, and the offers of hardware also. I can actually only handle the 4 machines I have here now but upgrading the 2 I have in use to the top end would be definitely the way to go, main one is almost there now of course, only thing left to do to it, would be a faster HT based CPU, possibly one with the new 1MB cache on it as well. And then move it's one over to server 2 which would handle less players on average I would say, *IF* that way comes about. As addicted points out, he and I have spoken about these things some time ago, and we do keep in touch with the pulse of things as much as possible to be pro-active about it.

The actual CPU use on crafting is fairly minimal overall, part of the testing was done in the early days with ATS use of 20 players all crafting at once, CPU never went over a plateau of about 12-15% CPU , what is happening constantly with all aspects of the server being invaded with all these nice folks, is handling information at various rates back and forth, so, for example if at any given time 20 players craft and the average of THEIR ping rates are say 1000 for argument sake, that reflects across the entire server. DB access is MINOR for a MySQL server setup, Salandra who is most definitely the most qualified person to comment on it, told me it hardly knows ANY of it is going on, however, my experience is, the most awesome drain is actually the log files and I am yet to find a way to stop ALL logging of NWNX activities, log files have grown in size on occasions to as much as 98MB that is HUGE log file sitting in memory access alongside the MOD being large also. If I could eliminate that size growth, that would help, along with moving the server vault possibly onto it's own drive, with the way players constantly log on and off for various reasons, it has to rebuild for example quest lists for a player each time they come on, it adds potentially 28 entries of 2-3 lines per player each time to log on. There are so MANY factors involved, and at the end of the day, we still will not completely win with a server program that was never made to do this, we'll just make it the best we can, and for some people, it will still not be enough .

More as we go,

Makz.

[ride]

Makzimia said:

Trying to preach to the converted ride? I am an old hand, I just like to make all things possible plain for all. I think I am covering my bases.

Makz.

no I just figure your like me, and when trying to fix something, sometimes forget about that simple rule of troubleshooting...

if you gys need co-lo stuff or a CVS server (to stop clobbering each others code changes) I can setup *any* of that up for you guys. and Makz I do have 1 1U Dell server system, it is nothing fancy, 600Mhz I believe and sounds like a cessna, but could be a good SQL backend system... I'm sure I'd be happy to "donate" it to our cause

[Sarovek]

Makzimia said:

If I could eliminate that size growth, that would help, along with moving the server vault possibly onto it's own drive, with the way players constantly log on and off for various reasons, it has to rebuild for example quest lists for a player each time they come on, it adds potentially 28 entries of 2-3 lines per player each time to log on.
Makz.

The guys over at Faerun-Nights have done that on our server. We are hosting 2 modules on it right now each with a different port and the only way to get 2 different server vaults was by writing a small application. Basanda was the person that made this possible, maybe you can talk to him on www.faerun-nights.com, just tell him a WOLF Admin (Major König) sent you and I'm pretty sure he would help you with that issue if help was needed :-)

[Makzimia]

I actually have my P3 933 here which could do that as well ride, but, I think an extra drive in the main server, and or one in server 2 may be the way to go, *IF* we go that way. In a LOT of ways, I want to see what 1.63 patch really does to performance.

Thanks Sarovek, but server vault seperation is actually as simple as a change in an ini file . Logging from NWNX is the issue I referred to, I can move a 20GB drive in and put it on server vault duties anytime I want really, server down for 20 mins tops. I wonder how much difference it would make moving the server vault over though.... other than obvious disk I/O which one would think the SATA with a 8MB cache would be barely blimping, more likely bad mem leaks etc per biowares admittance.

Makz.

[addicted2rpg]

Sarovek said:

different port and the only way to get 2 different server vaults was by writing a small application.

I don't think we want two different server vaults. Makz brought up the shared drive, something I missed, but that would be the solution (along with the shared DB).

Phoenix- good point about "chopping" Fredian up into little pieces, but I don't think it will happen that way sorry. It would be a lot of work to chop up the world like that, and the frequent server portaling may annoy players, although that is something I would have to test to be certain of, but my hunch is that it would, because NWN runs some extra procedures beyond the area load: it saves your char, connects to the new server, authenticates as your char (reloads from the vault), checks to make sure it is legal (which takes surprisingly long, btw...), and then loads you into the other server. I'm not a portal master yet, but I am sure I will be after researching it, but there may even be the issue of login scripts firing too, rebuilding your journal on the other server, etc... Ideally, we would want to bottleneck all portaling to one place. Maybe even do the underdark as the alternate theme just to say its too deep for the kehaar books to reach to the surface so we don't have to update them (it sounds like a cop-out, but it does seem to have a familiar ring with FR magic. Why should anything be ranged infinite?). I just so happen to have a certain functionality available to make areas un-kehaar-able... It was developed for the jails initially, but was fortunately rather flexible. We can add a little module variable on its startup deadkehaar No need to specify each area.

I'm seriously thinking that if Makz wants to do this, we would start off with a 1 area mod called blahblah or the crimson depths or whatever, and just try getting the portal to there working, update all the scripts, etc.. test the database, try jailing someone from there, etc.. Once the little 1 area mod falls into suitable compliance, Makz & other area contributors can begin development on it with ease.

[Phoenix]

Just looking for different ways to split the server to maintain consistant RP. If it takes all that to portal, frequent loading may not be desirable. Other ways wind up with a 'left-half' and 'right-half' of fredian, otherwise you won't get enough players on one side to help the other.

IE if you made certian areas server2 what's to guarantee enough players will be in the areas of server 2 to help server1, unless it's a main point of intrest. If the portal splits point of interest (like splitting styne and castille somwhere around the farms) then there will be RP issues (people can't contact other players on the other side).

It's less a point though, since crafting isn't as intensive as I though. It would reduce the total players on the main server and that would help...

Anyways, there really isn't much of a choice here. We either grow by portaling or we don't by limit the number of players (already done). Some upgrades will help the main server, but it's already quite beefy. If NWN was a multithreaded app., it would be different, but it isn't.

[addicted2rpg]

Correct!

S2 has to be as interesting for S1. And that will be a problem we'll deal with when we can The master of area design himself though can probably make S2 *very* interesting *makes a bunch of promises for Makz to fulfill hehehe*. Of course, we havn't decided on making S2 yet, not without the big word from the chief. We haven't even decided on a theme for it yet. I just threw the underdark as an example.

It is all very preliminary, and will take a lot of time.

And Correct Again!

Its either get fat like a blimp and stuck, or move to new lands

It is just a hunch though, but I think this will be the direction we will eventually go. It just seems to be what the other servers do. Even if our numbers dip down, I'd still be for it, because new people join at a faster rate as a function of lag, speed even... if it is not laggy, just having things faster than the other guy brings'em out.

[MitzaVolchenko]

Cut our numbers down for now, and work on scenarios for the long run. I also agree that pulling the gamespy listing is not a bad thing for now.
I have soem notions, but at the moment I am mildly doped and not thinking at my best.