Tweet
I am not trying to sell a product here but rather I am trying to find out if there is sufficient interest to complete a program.
I have received quite a few Ts2ki Unfrag Reports during the license process for Ts2ki Unfrag which show the sizes and numbers of Ts2ki Global Server (GS) files and although many of these reports show the sizes of the normal 10 GS files to be relatively close to each other there are quite a few which vary widely in size. The most abnormal report I have seen so far had 22 GS files with sizes all the way between a high of 3.983 GB (the absolute maximum is 4.00 GB) and a low of 0.004 GB. This did not look like an optimum situation so I decided to see if there was anything that could be done. As a result, I have the following Proof of Concept which should give the User almost complete control of GS Average File Size and Number of GS files.
The way the standard Ts2ki program works is that whenever symbol data is added that symbol is initially assigned to one of the normally 10 GS files and the symbol and data remain in that GS file forever under normal situations. The implication here is that you can never have more than 4 GB of data for any GS symbol even if there is only one symbol in the GS file. Actually the maximum data size value is about 6 KB less than 4 GB but that is extremely close to 4 GB.
The concept I have generated in order to overcome this file relative size problem is:
1. To copy the entire GS PDS folder somewhere on the PC or external drive as a backup and date it so that we could have more than a single backup if desired. So, the current GS PDS folder copy would become "PDS 2009-08-11" or whatever the actual date is.
2. Delete all the 0000*.DAT and 0000*.IDX files from the current GS PDS folder.
3. Go through the backup PDS folder GS DAT files and extract symbol data for from the backup DAT files for symbols which would most closely match a User selected maximum GS file size and then write a new GS file starting with GS file 00000001.dat and continuing until all data was extracted from the backup files - 00000002.dat, 00000003.dat etc.
4. Start the Ts2ki GS and allow it to create index files (IDX) for each of the DAT files.
As a Proof of concept I have done the following this morning and just to make the proof of concept process easier I just added a fixed number of symbols to each new GS file. Picking the correct symbols in order to get the best file size results is only a detail.
I started with a GS Symbol List which contained 109 symbols and initially had the standard 10 GS files which produced the following image of two sample symbol charts. I added a trend line on one chart and a comment on the other because someone had asked whether altering the GS files would affect lines and comments.
Then, for this Proof of Concept, rather than copy the entire PDS folder for backup data I just merged all the data from all GS files into GS File 00000001.dat and then distributed a fixed number of symbols and the corresponding data to the other files - 00000002, 00000003 etc until all data was extracted.
Re-distributing the data from the Initial 10 GS files to 50 GS files produced the following:
Re-distributing the data from the 50 GS files to 99 GS files produced the following:
The resulting 99 GS File List is shown in the image:
Re-distributing the data from the 99 GS files back to 5 GS files produced the following:
The resulting 5 GS File List is shown in the image:
Re-distributing the data from the 5 GS files back to 1 GS file produced the following:
The resulting 1 GS File List is shown in the image:
Just to show the actual Time Sequence when operations were performed the following Time Line was included:
The File Lists show large size differences between the GS files but this is only because this Proof of Concept did not pick the symbols which would end up with the best GS file sizes.
This program would also allow the user to change GS file sizes to possibly some better size. Maybe 5 files of 2 GB in size would be better than 10 files of 1 GB in size or possibly another combination would work better.
It seems that with a bit of work this program could give the user a lot of power and if there is sufficient interest I will complete it.
All comments would be appreciated.
Best regards,
~Bob
I have received quite a few Ts2ki Unfrag Reports during the license process for Ts2ki Unfrag which show the sizes and numbers of Ts2ki Global Server (GS) files and although many of these reports show the sizes of the normal 10 GS files to be relatively close to each other there are quite a few which vary widely in size. The most abnormal report I have seen so far had 22 GS files with sizes all the way between a high of 3.983 GB (the absolute maximum is 4.00 GB) and a low of 0.004 GB. This did not look like an optimum situation so I decided to see if there was anything that could be done. As a result, I have the following Proof of Concept which should give the User almost complete control of GS Average File Size and Number of GS files.
The way the standard Ts2ki program works is that whenever symbol data is added that symbol is initially assigned to one of the normally 10 GS files and the symbol and data remain in that GS file forever under normal situations. The implication here is that you can never have more than 4 GB of data for any GS symbol even if there is only one symbol in the GS file. Actually the maximum data size value is about 6 KB less than 4 GB but that is extremely close to 4 GB.
The concept I have generated in order to overcome this file relative size problem is:
1. To copy the entire GS PDS folder somewhere on the PC or external drive as a backup and date it so that we could have more than a single backup if desired. So, the current GS PDS folder copy would become "PDS 2009-08-11" or whatever the actual date is.
2. Delete all the 0000*.DAT and 0000*.IDX files from the current GS PDS folder.
3. Go through the backup PDS folder GS DAT files and extract symbol data for from the backup DAT files for symbols which would most closely match a User selected maximum GS file size and then write a new GS file starting with GS file 00000001.dat and continuing until all data was extracted from the backup files - 00000002.dat, 00000003.dat etc.
4. Start the Ts2ki GS and allow it to create index files (IDX) for each of the DAT files.
As a Proof of concept I have done the following this morning and just to make the proof of concept process easier I just added a fixed number of symbols to each new GS file. Picking the correct symbols in order to get the best file size results is only a detail.
I started with a GS Symbol List which contained 109 symbols and initially had the standard 10 GS files which produced the following image of two sample symbol charts. I added a trend line on one chart and a comment on the other because someone had asked whether altering the GS files would affect lines and comments.
Then, for this Proof of Concept, rather than copy the entire PDS folder for backup data I just merged all the data from all GS files into GS File 00000001.dat and then distributed a fixed number of symbols and the corresponding data to the other files - 00000002, 00000003 etc until all data was extracted.
Re-distributing the data from the Initial 10 GS files to 50 GS files produced the following:
Re-distributing the data from the 50 GS files to 99 GS files produced the following:
The resulting 99 GS File List is shown in the image:
Re-distributing the data from the 99 GS files back to 5 GS files produced the following:
The resulting 5 GS File List is shown in the image:
Re-distributing the data from the 5 GS files back to 1 GS file produced the following:
The resulting 1 GS File List is shown in the image:
Just to show the actual Time Sequence when operations were performed the following Time Line was included:
The File Lists show large size differences between the GS files but this is only because this Proof of Concept did not pick the symbols which would end up with the best GS file sizes.
This program would also allow the user to change GS file sizes to possibly some better size. Maybe 5 files of 2 GB in size would be better than 10 files of 1 GB in size or possibly another combination would work better.
It seems that with a bit of work this program could give the user a lot of power and if there is sufficient interest I will complete it.
All comments would be appreciated.
Best regards,
~Bob
