JTools Table of Contents

Project Location: \JTools_StateFileGenerator

NxCore is a whole market-feed processing system. As such, it must process tapes from the start of the day in order to properly populate and maintain internal data structures through all corrections and exchange messages. However, NxCore capable of saving it's internal state at any time and restarting from that point using the saved state Understanding how to effectively generate and use NxCore State file can unlock many doors, specifically those relating to jumping into the tape at any given point in time. Some of the many possibilities include:
  • Fast access to all tick data. Using multiple timed state files jumping to and from different times in any given tape becomes trivial.
  • Backup for an application that may have crashed and needs to re-enter the real time buffers as quickly as possible at start-up.

In fact, the two most common question we hear from NxCore users both regard how to jump to specific points in time when processing from the NxCore API:
  • "I want to jump into the real time buffers right away without running the tape from the beginning of the day."
  • "I want to jump to any point in time quickly and begin the stream there."

Even advanced NxCore users may not be aware of the full potential State Files offer.

The State Writer Application is a revision of the JDStateWriter application, which I made available to users some time ago. Since the initial version however, I have modified the application for a numbers of various tasks as well as improve the code in general. I have recently been converted it to the JTools framework and I have included every option I have ever needed with regards to automatic NxCore state file generation.

  • Prime Time Trading - The frequency (in minutes) of when to save state files during the hours of 9:30 AM EST to 16:00 PM EST.

  • FormT Trading - The frequency (in minutes) of when to save state files during the hours of 4:00 AM EST to 9:30 AM EST and 16:00 PM EST to 20:00 PM EST.

  • Non Prime Time - The frequency (in minutes) of when to save state files during the hours of 0:00 AM EST to 04:00 AM EST and 20:00 PM EST to 24:00 PM EST.

  • Weekends - The frequency (in minutes) of when to save state files on weekends (if weekends are enabled)

  • Save Time Offset - Some applications will prefer to have the state file saved seconds before the actual save time. For instance, if a save is approaching at 10:00:00, the developer may wish to save the actual state at 09:59:45, for a 15 second lead into the desired time frame. Use the Save Time Offset field to set a lead time if desired.

  • Start Time - The start time that all messages are to be filtered by. State files will not be generated before this time.

  • Stop Time - The stop time that all messages are to be filtered by. State files will not be generated after this time.

  • State File Location - Folder location where the app is to generate state files at. If this field if left blank, the default sub-folder of JTools_StateFiles will be created (in the folder the application runs in) and state files will be created in the sub-folder.

  • Archive File Location - Folder location where older state files are to be copied and moved each day, if set to do so.

  • Spawn StateSpinner on -12 Error- If set will launch the state-spinner application on -12 errors. See below for more information.

  • Overwrite Existing Files- If set, and state files of the same name exist, the current state files are overwritten.

  • Process Saturay - Enabled state file generation on Saturday.

  • Process Sunday - Enabled state file generation on Sunday.

  • Save Files by TimeStamp + Save by Date- If set state files are save in both date/timestamped filenames and a single dated filename with the most recent state (see below).

  • Save Files by TimeStamp - If set, files are saved with both the date and time in the filename. Multiple files will be created each day.

  • Save File by Date - If set, the current date is used to set the most recent state file. The state file will be updated at every interval.

  • Copy Previous Day's Files to Archive Location - If set, all files in the current state file folder will be copied to the archive location when a new tape is run.

  • Move Previous Day's Files to Archive Location - If set, all files in the current state file folder will be moved to the archive location when a new tape is run.

  • Delete Previous Day's Files - If set, all files in the current state file folder will be deleted when a new tape is run.

  • Starting Address - Memory location where NxCore if to start when ProcessTape is called. This is an advanced setting and I recommend leaving it set to the default of 0.

    NOTE: As of NxCore version 2.6.26 this parameter is obsolete, and NxCore will now automatically re-base the memory location. We have kept the field here only for backwards compatibility.

Having state file generation done in a separate, stand-alone application such as this, as opposed to a state file generator imbedded into each NxCore application offers several benefits:
  • State file generation is not free. There is a performance hit and if state files are being generated frequently, this hit may an impact on the application. By generating the state files in an application who's job is to do nothing but create state file, this hit can be eliminated from all other NxCore applications running on the machine.

  • If the goal is to provide a recovery point in case an application crashes and the application does indeed crash, the state file generator will keep generating current state files. Regardless of how long the application is down or takes to restart, it will always have a recent point in time to start NxCore from a state file.

  • Avoids the need for multiple state file locations if multiple applications are using state files on the same machine (as duplicate file write collisions will not be an issue).

All of the JTools applications have the ability to load and start from state files. Simply choose the state file as opposed to a full NxCore history file when choosing historical tapes. The state files the application generates will all end with the file extension ".nxs" as opposed to ".nxc" (the extension of NxCore historical tapes).

One Caveat

Because the State file write mechanism is a separate thread. there is one potential hitch when writing State Files; trying to generate a state file while another state file is still in the process of being created. This generates a -12 error (NxAPIERR_SAVE_STATE_IN_PROGRESS). There is no clean way to recover from the error and generate the state file you were intending to within the same NxCore thread.

The problem is rarely encountered when running the State Generator in real time (when it is in the real time buffers) and time is moving rather slow. To take this to an extreme however, consider loading from a historical weekend tape and running it as fast as possible. There is such little data that you could get to the end of the day in one or two minutes (if that). If you were trying to save a state file every 15 minutes (minutes on the tape) in this manner, you will generate -12 errors on multiple state file generation attempts. There has simply not been enough elapsed time for the last state file to be saved completely and the system will not generate another until the save state thread is complete.

If during these situations, it is critical for you to generate every state file as expected, there are 2 options. First, you could build more threads into the main application and relaunch multiple instances to regenerate the state files . Second, you could off-load the task to another application.

I chose the second of these for the state generator application, for a number of reasons:
  • The method works *very* well.

  • The seperate application could be spawned by multiple applications.

  • To demonstrate off-loading NxCore tasks to another application or process entirely. This can open up other doors. While the information flow in the State File Generator is one way, if the spawned process had two way communication to the application that spawned it, a world of possibilities exist.
Below is a screen shot of the State Generator application spawning the "JTools_StateSpinner" application on -12 errors. The extreme case was used to demonstrate it (a weekend, late afternoon and starting the tape from the begging going full blast and finishing in approx 2 minutes):

The code for the JTools_StateSpinner application is imbedded in the JTools_StateFileGenerator folder as \JTools_StateFileGenerator\JTools_StateSpinner\

JTools Table of Contents