|JTOOLS STATE FILE GENERATOR
JTools Table of Contents
Project Location: \JTools_StateFileGenerator
NxCore is a whole market-feed processing system. As such, it must process tapes
from the start of the day in order to properly populate and maintain internal
data structures through all corrections and exchange messages. However, NxCore
capable of saving it's internal state at any time and restarting from that
point using the saved state Understanding how to effectively generate and use
NxCore State file can unlock many doors, specifically those relating to jumping
into the tape at any given point in time. Some of the many possibilities
- Fast access to all tick data. Using multiple timed state files jumping to
and from different times in any given tape becomes trivial.
- Backup for an application that may have crashed and needs to re-enter the
real time buffers as quickly as possible at start-up.
In fact, the two most common question we hear from NxCore users both regard how
to jump to specific points in time when processing from the NxCore API:
- "I want to jump into the real time buffers right away without running
the tape from the beginning of the day."
- "I want to jump to any point in time quickly and begin the stream
Even advanced NxCore users may not be aware of the full potential State Files
The State Writer Application is a revision of the JDStateWriter application,
which I made available to users some time ago. Since the initial version
however, I have modified the application for a numbers of various tasks as well
as improve the code in general. I have recently been converted it to the JTools
framework and I have included every option I have ever needed with regards to
automatic NxCore state file generation.
- Prime Time Trading - The frequency (in minutes) of when to save
state files during the hours of 9:30 AM EST to 16:00 PM EST.
- FormT Trading - The frequency (in minutes) of when to save state
files during the hours of 4:00 AM EST to 9:30 AM EST and 16:00 PM EST to 20:00
- Non Prime Time - The frequency (in minutes) of when to save state
files during the hours of 0:00 AM EST to 04:00 AM EST and 20:00 PM EST to 24:00
- Weekends - The frequency (in minutes) of when to save state files on
weekends (if weekends are enabled)
- Save Time Offset - Some applications will prefer to have the state
file saved seconds before the actual save time. For instance, if a save is
approaching at 10:00:00, the developer may wish to save the actual state at
09:59:45, for a 15 second lead into the desired time frame. Use the Save Time
Offset field to set a lead time if desired.
- Start Time - The start time that all messages are to be filtered by.
State files will not be generated before this time.
- Stop Time - The stop time that all messages are to be filtered by.
State files will not be generated after this time.
- State File Location - Folder location where the app is to generate
state files at. If this field if left blank, the default sub-folder of
JTools_StateFiles will be created (in the folder the application runs in) and
state files will be created in the sub-folder.
- Archive File Location - Folder location where older state files are
to be copied and moved each day, if set to do so.
- Spawn StateSpinner on -12 Error- If set will launch the
state-spinner application on -12 errors. See below for more information.
- Overwrite Existing Files- If set, and state files of the same name
exist, the current state files are overwritten.
- Process Saturay - Enabled state file generation on Saturday.
- Process Sunday - Enabled state file generation on Sunday.
- Save Files by TimeStamp + Save by Date- If set state files are save
in both date/timestamped filenames and a single dated filename with the most
recent state (see below).
- Save Files by TimeStamp - If set, files are saved with both the date
and time in the filename. Multiple files will be created each day.
- Save File by Date - If set, the current date is used to set the most
recent state file. The state file will be updated at every interval.
- Copy Previous Day's Files to Archive Location - If set, all files in
the current state file folder will be copied to the archive location when a new
tape is run.
- Move Previous Day's Files to Archive Location - If set, all files in
the current state file folder will be moved to the archive location when a new
tape is run.
- Delete Previous Day's Files - If set, all files in the current state
file folder will be deleted when a new tape is run.
- Starting Address - Memory location where NxCore if to start when
ProcessTape is called. This is an advanced setting and I recommend leaving it
set to the default of 0.
NOTE: As of NxCore version 2.6.26 this parameter is obsolete, and NxCore will
now automatically re-base the memory location. We have kept the field here only
for backwards compatibility.
Having state file generation done in a separate, stand-alone application such
as this, as opposed to a state file generator imbedded into each NxCore
application offers several benefits:
- State file generation is not free. There is a performance hit and if state
files are being generated frequently, this hit may an impact on the
application. By generating the state files in an application who's job is to do
nothing but create state file, this hit can be eliminated from all other NxCore
applications running on the machine.
- If the goal is to provide a recovery point in case an application crashes
and the application does indeed crash, the state file generator will keep
generating current state files. Regardless of how long the application is down
or takes to restart, it will always have a recent point in time to start NxCore
from a state file.
- Avoids the need for multiple state file locations if multiple applications
are using state files on the same machine (as duplicate file write collisions
will not be an issue).
All of the JTools applications have the ability to load and start from state
files. Simply choose the state file as opposed to a full NxCore history file
when choosing historical tapes. The state files the application generates will
all end with the file extension ".nxs" as opposed to ".nxc"
(the extension of NxCore historical tapes).
Because the State file write mechanism is a separate thread. there is one
potential hitch when writing State Files; trying to generate a state file while
another state file is still in the process of being created. This generates a
-12 error (NxAPIERR_SAVE_STATE_IN_PROGRESS). There is no clean way to recover
from the error and generate the state file you were intending to within the
same NxCore thread.
The problem is rarely encountered when running the State Generator in real time
(when it is in the real time buffers) and time is moving rather slow. To take
this to an extreme however, consider loading from a historical weekend tape and
running it as fast as possible. There is such little data that you could get to
the end of the day in one or two minutes (if that). If you were trying to save
a state file every 15 minutes (minutes on the tape) in this manner, you will
generate -12 errors on multiple state file generation attempts. There has
simply not been enough elapsed time for the last state file to be saved
completely and the system will not generate another until the save state thread
If during these situations, it is critical for you to generate every state file
as expected, there are 2 options. First, you could build more threads into the
main application and relaunch multiple instances to regenerate the state files
. Second, you could off-load the task to another application.
I chose the second of these for the state generator application, for a number
Below is a screen shot of the State Generator application spawning the
"JTools_StateSpinner" application on -12 errors. The extreme
case was used to demonstrate it (a weekend, late afternoon and starting the
tape from the begging going full blast and finishing in approx 2 minutes):
- The method works *very* well.
- The seperate application could be spawned by multiple applications.
- To demonstrate off-loading NxCore tasks to another application or process
entirely. This can open up other doors. While the information flow in the State
File Generator is one way, if the spawned process had two way communication to
the application that spawned it, a world of possibilities exist.
The code for the JTools_StateSpinner application is imbedded in the
JTools_StateFileGenerator folder as
JTools Table of Contents