Precaching attempts

[17324559]

Ok, @**[g4borg](< base_url >/index.php?/profile/239042-g4borg/ “Go to g4borg’s profile”) **made a compiled version of this (thx, mate!). I added the version to preload the whole Data folder, so:

 - if you have less than 12GB of RAM, try this version:

[SCPrecache.zip](< base_url >/applications/core/interface/file/attachment.php?id=15697)

• If you have 12GB or more and you want to preload the whole Data folder:

[SCPrecacheAll.zip](< base_url >/applications/core/interface/file/attachment.php?id=15698)

 

In both cases: unpack the tool into your game folder and double click on it. It will display command window for some time and then it will close it. You are good to go. 

If in game you will go to different mode than co-op or PVP and you’ll want to play some PvP/co-op, it may be necessary to rerun it (you don’t need to quit the game).

 

Old version:

Very crude selective precaching for windows.

Just get the file to your game folder, unpack it and run precache.bat, then run the game. Less apps open the better!

[precache.zip](< base_url >/applications/core/interface/file/attachment.php?id=15678)

 

 

[5173541]

6 minutes ago, niripas said:

Very crude selective precaching for windows.

Just get the file to your game folder, unpack it and run precache.bat, then run the game. Less apps open the better!

[precache.zip](< base_url >/applications/core/interface/file/attachment.php?id=15678)

 

 

Thanks!

[ORCA1911]

I tried the batch file, i dont know if i did it right but nothing hapened

[17324559]

Try this:

• freshly booted system (no browsers launched, nothing)

• unpack this in your game folder

• double click

• wait for it to finish (with your disks it can take a while)

• run the game

• launch co-op or pvp

• check if the freezing still occurs

 

for verification purposes, launch only pvp or coop. If you’ll launch to any other mode, precached data will be overwritten

as I said it’s crude and slow, but seems to be working. 

If anyone knows cpp for windows, I can gladly explain what has to be done - it will be MUCH faster done as a compiled program (10-20 seconds)

[3522870]

so you mean reading all files once into nowhere before you start it is some sort of precache?

how is that supposed to help? I mean if it does help, it would surprise me greatly - nah i believe this is possibly a placebo.

 

what needs to be done is fixing the code of the game, by e.g. to not read the data in too large buffers and not all at once, and possibly do that in a thread async.

and the coder should try to use a system without SSD or reproduce it at least

 

all this of course are just guesses, stuff i would try first, point is, it has to be fixed from the inside.

[17324559]

1 hour ago, g4borg said:

so you mean reading all files once into nowhere before you start it is some sort of precache?

how is that supposed to help? I mean if it does help, it would surprise me greatly - nah i believe this is possibly a placebo.

 

what needs to be done is fixing the code of the game to not read the data in too large buffers and not all at once, and possibly do that in a thread async.

 

and the coder should try to use a system without SSD.

not nowhere. The whole idea is to get those most common files mapped to Standby memory. 

How is that supposed to help? Very simple. When a player joins a game with something that is not already on battlefield, game engine has to load that data - models, textures etc etc, then unpack it and access that data. As people noticed that creates a noticeable lag, which is smaller, the faster I/O of the disk is. So it got me thinking. If it would be possible to actually put those files in RAM before they will be accessed via the game engine. I read few things about how Windows file access works (sorry, I know very little about Windows OS) and as in other unix-kernel OS it caches the file it reads into memory (called Standby memory in Windows) in case that file will be accessed in the near future. 

 

Quote

how is that supposed to help? I mean if it does help, it would surprise me greatly - nah i believe this is possibly a placebo.

Very simple. People with conventional HDDs have I/Os in range of milliseconds, so loading files needed and accessing them can take over 2 seconds. Mapping those files into standby memory that time is reduced to microseconds and because file is already in the memory, there is no need to access the HDD at all. 

Simple explanation is here (via https://www.techrepublic.com/article/how-to-use-windows-10s-resource-monitor-to-track-memory-usage/ )

Quote

(Standby memory) contains pages that have been removed from process working sets, but that are still linked to their working sets. The Standby list is basically a cache, but memory pages in the list are prioritized in a range of 0-7 (with 7 being the highest).

A page related to a high priority process will receive a high priority level in the Standby list. For example, processes that are Sharable will be a high priority and pages associated with them will have the highest priority in the Standby list.

If a process needs a page that is associated with the process, and that page is now in the Standby list, the memory manager will return the page to that process’ working set. However, all pages on the Standby list are available for memory allocation requests from any process. When a process requests more memory and there’s not enough in the Free list, the memory manager checks the page’s priority. It will then take a page with a low priority from the Standby list, initialize it, and allocate it to that process.

 

Placebo? Hmm. It’s much faster than any RAMdisk or SSD. Does it work? 

 

Quote

what needs to be done is fixing the code of the game to not read the data in too large buffers and not all at once, and possibly do that in a thread async.

I’m glad you know what needs to be done, however I don’t have access to the source code. I had idea this morning and did a proof of concept workaround. I have no idea how to code in Windows and basically the whole issue got me almost 3 hours to figure out how to get those files mapped to the memory without using any Unix tools  ”) (and no, copy filename >NUL doesn’t work - copy detects the second argument is NUL and just accesses the file, without reading it). Faster approach would be of course using Windows API ReadFileEx function (yep, google is my friend) - but as I said, I have no idea how to code on Windows platform, and since this whole endeavour started this morning at 9AM, I have no intention of learning that. In another thread  I uploaded the tool that reads all files in the directory and the whole data folder of this game took about 15 seconds to be memory mapped from mechanical HDD. I fully agree that this could be done while i.e. waiting in a pvp/coop queue (or during flight in OS - preload sectors of all gates available from that sector), but at  this moment this is what we have. 

 

BTW: I can be completely wrong and “freeze while join in progress” can be caused by something else. That’s why I am not posting this as a solution in general part of the forum.

 

[3522870]

 

36 minutes ago, niripas said:

I’m glad you know what needs to be done,

i corrected the sentence, to reflect, that i am of course without source code acces not sure what is to be done exactly, but sure it has to be done on the engine side of things, if done properly. but i guess we are on the same page there. however that should eliminate the sarcastic tone.

 

36 minutes ago, niripas said:

not nowhere.

36 minutes ago, niripas said:

(and no, copy filename >NUL doesn’t work - copy detects the second argument is NUL and just accesses the file, without reading it)

>NUL is just the same as > /dev/null, which is what i meant to nowhere

basicly it tells type to read the file but dont output it to the screen

the second nul does the same for errors

 

36 minutes ago, niripas said:

It’s much faster than any RAMdisk or SSD

well if that standby memory thing is true, this basicly is a ram disk

just a very unreliable one.

and i did find hintsthat such a behaviour indeed exists since win7. good find. (they also reference a tool there called RamMAP from sysinternals for that stuff)

well then its basicly not a placebo, it might just not work for everyone and in every situation.

 

36 minutes ago, niripas said:

I have no idea how to code in Windows

reading a file should be the same, if type works, any read should work, it’s probably why they implemented it low level.

but you could forward that msdn command to the dev team and tell them to fix asset loading properly.

 

especially since the loading lags do happen even after a few hours of play unfortunately, aswell.

 

36 minutes ago, niripas said:

whole endeavour started this morning at 9AM

you are a hero.

 

but indeed, this trick might be handy for many things.

[17324559]

4 minutes ago, g4borg said:

 

i corrected the sentence, to reflect, that i am of course without source code acces not sure what is to be done exactly, but sure it has to be done on the engine side of things, if done properly.

 

>NUL is just the same as > /dev/null, which is what i meant to nowhere

basicly it tells type to read the file but dont output it to the screen

the second nul does the same for errors

 

well if that standby memory thing is true, this basicly is a ram disk

just a very unreliable one.

and i did find hints that such a behaviour indeed exists since win7. good find.

well then its basicly not a placebo, it might just not work.

 

 

especially since the loading lags do happen even after a few hours of play unfortunately, aswell.

 

 

yeah, my first approach was just to do equivalent

cp filename /dev/null

but of course windows copy is too smart to do this and only header was cached (try to copy 4GB file to NUL it will take exactly the same amount of time as few kB - and darn it’s “copy filename NUL”, >NUL just redirects output, which as you probably noticed, I am doing with type in that batch file).

 

Standby memory thing is true - it’s the same thing as in Linux or other Unices. And yes - it is unreliable. It keeps that data as long as it doesn’t need RAM for anything else. Also it is much faster than creating RAMdisk (you are skipping the driver thingie, block devices etc etc). 

Quote

reading a file should be the same, if type works, any read should work, it’s probably why they implemented it low level.

but you could forward that msdn command to the dev team and tell them to fix asset loading properly.

 

Heck - type is very slow. That ReadFileEx  function is basically 20 times faster as it doesn’t need to generate output in the ascii form and then discard it 

Quote

you are a hero.

 

thank you 

[3522870]

i think on unix this is done by the filesystem, i guess, at least ext does a lot of smart caching.

 

yes i saw it now, that rammap tool even shows me stuff i opened yesterday from my pc over the network. quite impressive. that tool should help seeing if the whole ordeal works.

 

they could add the same procedure in the game at startup and add a ticker in settings to perform this task internally once, for a lazy fix. they might like a lazy fix solution.

 

i did a version in .Net, source is attached, but i am not sure if its faster (it is faster the second time )

same usage, optional argument to exe is the absolute path.

 

SCPrecache.zip

[ORCA1911]

Well i took the nab way and went with amd ramdisk thingy, created a 3GB ramdisk and loaded everything except maps, took a nice chunk out of the annoying join lags. Orca is satisfied. 

[17324559]

9 hours ago, g4borg said:

i think on unix this is done by the filesystem, i guess, at least ext does a lot of smart caching.

 

yes i saw it now, that rammap tool even shows me stuff i opened yesterday from my pc over the network. quite impressive. that tool should help seeing if the whole ordeal works.

 

they could add the same procedure in the game at startup and add a ticker in settings to perform this task internally once, for a lazy fix. they might like a lazy fix solution.

 

i did a version in .Net, source is attached, but i am not sure if its faster (it is faster the second time )

same usage, optional argument to exe is the absolute path.

 

[SCPrecache.zip](< base_url >/applications/core/interface/file/attachment.php?id=15683)

It’s a lot faster than type  Thx!

Well - it can be even done in the game - if there is memory available as soon as player hits the queue, preload/precache models accordingly to the ship ranks in slot ”)

[5173541]

It definitely works, but not for very long. Developers should do this on their own, but I got a feeling that [niripas] knows more about the coding than the developers. No pun intended, of course.

[17324559]

1 minute ago, Koromac said:

It definitely works, but not for very long.

You can just run this small utility every time you want to do some coop or pvp. However the more permanent solution (but with more hassle) would be to put those files in RAMdisk - then they will not be purged by other tasks. However I have no idea how to do this in Windows (ORCA did it, so… ORCA?).

[ORCA1911]

Well, its quite easy and i guess lasts longer than precache you guys are using, the current downside one can have is crammed ram lol

For the time being i went with the most popular tool AMD Radeon™ RAMDisk. You have only 4GB option for free tho but what i did is, i just loaded everything except maps and that took 3GB of ram space, it cut down the join lag by a lot for me but it still exists in a small measure that i can live with, much better than total freeze anyway.

 

Once you download it, it’s pretty straightforward, you create a ramdisk, start it, a window will open and it will look like an emulated HDD/SSD so you just copy-paste the files from data folder in there and start your game. You should notice the improvement.

[13484520]

I see two programs to put game into ramdisk. I think need put whole game folder  into ramdisk. First program is RAMDisk (link). The only problem, that it is free only with 4GB ramdisk then will be paid. But the game folder size is about 6.6 GB. There is another program, ImDisk. It permit 10GB ramdisk and free. Description link. I think this only work with the launcher installed program, because of the Steam has huge folder. And need to edit the shortcut properties with new ramdisk letter.

[3522870]

3 hours ago, niripas said:

It’s a lot faster than type  Thx!

good to hear. np.

for this operation i do expect it to run as fast as a c++ app, and using c# should eliminate the problem of 32/64 bit.

luckily i had a standalone dev environment from space engineers modding, because i also just code on linux - and c# is only for unity3d.

 

2 hours ago, Koromac said:

definitely works, but not for very long.

my result was, that i had to rerun the precache after a few hours of work on the pc, but as long as star conflict was running and i did not do anything else besides it, the cached files stayed where they are. so i doubt, they will be “uncached” while only SC is running. It does depend on your amount of RAM of course, I doubt its useful below 16 gigs.

As said, I even saw a 400 MB file in RAM from something I definitely used the day before when i first started it up. the other tool we talked about in the thread allows to actually see which files are in that magic memory standby mode.

 

in the game it worked. however, i did still feel the first joinlag slightly, so probably there was still a file not on the list. but it was barely noticable.

i had oregyen on the other side, who had to report his join lags, and he reported a lot more, i only felt the first, and a lot more slightly than usual (half a second, and i am very very sensitive to stuff like this)

so it definitely works.

 

as said, it depends on how much ram you guys have. and its possible to run the script actually after you entered SC aswell.

 

with low ram, both options are less of an option. the ramdisk will reduce your available memory, and you have to load a lot more on it than you actually need. the cache will be emptied a lot earlier than needed.

 

and while precaching the data is helpful, i am still rather on the side of not doing it like this in the game, but rather fix the asset loading mechanism to make slight delays between ops, do the whole thing serially (file after file), use one dedicated thread, preallocate players, etc. so anything to reduce the sudden extreme loading of data when people join. since joining players have a countdown, there is more than enough time.

[ORCA1911]

On a second though, to truly avoid the join in lag, the best solution would be to create a 7gig ramdisk and load the whole game there, the precache files you guys provided didnt do anything for me sadly, and the stuff i tried out is also in the not very useful category.

[3522870]

whats your ram / os?

do you have multiple harddrives?

do you use a laptop?

did you try RamMapto see if the files are in the mapfile category?

 

and of course, the devs should assign a dev to fix the engine. all these attempts are fixes from the wrong vector. but you guys sit closer to that solution.

[ORCA1911]

8gig ddr3 classic ram 1333, one drive, desktop, nothing changed in my ram regardless of how i executed the exe, im trying the aforementioned thing maybe it works, BSOD here i come ;D

[3522870]

with 8 gig having a 7 gig ramdisk is out of the question, as you will have 1 gig left for system, which means, you will basicly slow down your pc even more as it tries to swap things in and out from your pagefile.

with one hd, you will have to endure all data coming from one disk, so any read operation by windows will also interfere.

you should always leave at least 4 gigs for your system.

 

if you are on windows previous to win7 you also unfortunately might not have that file mapping feature.

 

i have 16; judging by the numbers, 8 is just not enough to use the feature effectively, 16 however is more than enough to even have free ram unused.