Important dataset preparation do's and don'ts

There are a few simple, but important, do's and don'ts to prepare your dataset for post-processing in StarTools.

Learning how to use a new application is daunting at the best of times. And if you happen to be new to astrophotography (welcome!), you have many other things, acronyms and jargon to contend with too. Even if you consider yourself an image processing veteran, there are some important things you should know. That is because some things and best practices play a bigger role in StarTools than in other applications. By the same token, StarTools is also much more lenient in some areas than other applications.

Most advice boils down to making sure your dataset is as virgin as possible. Note that doesn't mean noise-free or even good, it just means you have adhered to all the conditions and best-practices outlined here, to the best of your abilities.

If you are new to processing

When learning how to process astrophotography images, the last thing you want to do, is learning all sorts of post-processing tricks and techniques, just to work around issues that are easily avoidable during acquisition or pre-processing. Fixing acquisition and pre-processing issues during post-processing, will never look as good, while you will also not learn much from this; whatever you learn and do to fix a particular dataset, is likely not applicable to the next.

Conversely, if your dataset is clean and well calibrated according to best practices, you will find workflows much more replicable and shorter. In short, it is just a much better use of your time and efforts! You will learn much quicker and you will start getting more confident in finding your personal vision for your datasets - and that is what astrophotography is all about.

If practical, try a divide & conquer strategy, focusing on areas of data acquisition, pre-processing, and post-processing separately and in that order. Be mindful that success in conquering one stage is important to be able to achieve success in the stage that immediately follows it.

If you are an image processing veteran

When we say StarTools requires the most virgin dataset you can muster, we really mean it! It means no procedures or modifications must be done by any other software - no matter how well-meaning. It means no gradient or light pollution removal, no color balancing, not even normalization (if not strictly necessary for outlier rejection), and no pre-compositing of the channels. Signal evolution Tracking - the reason why StarTools achieves objectively better results than other software - absolutely requires it.

Checklist

• Make sure your dataset is as close to actual raw photon counts as possible.
  
• Make sure your dataset is linear and has not been stretched (no gamma correction, no digital development, no levels & curves)
  
• Make sure your dataset has not been normalised (no channel calibration or normalisation) unless unavoidable due to your chosen stacking algorithm
• Make sure all frames in your dataset are of the same exposure length and same ISO (if applicable)
  
• Make sure your dataset is the result of stacking RAW files (CR2, CR3, NEF, ARW, FITS, etc.) and not lossily compressed or low-bit depth formats (e.g. not JPEGs or PNGs).
  
• Make sure no other application has modified anything in your dataset; no stretching, no sharpening, no gradient reduction, no normalisation
  
• If you can help it, make sure your dataset is not color balanced (aka "white balanced"), nor has had any camera matrix correction applied
• Flats are really not optional - your dataset must be calibrated with flats to achieve a result that would be generally considered acceptable
  
• Dithering between frames during acquisition is highly recommended (a spiralling fashion is recommended, and if your sensor is prone to banding, you will want to use larger movements)
• If you use an OSC or DSLR, choose a basic debayering algorithm (such as bilinear or VNG debayering) in your stacker. Avoid "sophisticated" debayering algorithms (Astro Pixel Processor's AAD excepted) meant for single frames and terrestrial photography like AHD or any other algorithms that attempt to reconstruct detail.
• If using a mono CCD/CMOS camera, make sure your channels are separated and not pre-composited by another program; use the Compose module to create the composite from within StarTools and specify exposure times where applicable.
• Make sure you use an appropriate ISO setting for your camera (see Recommended ISO Settings for DSLR cameras section)
• If stacking multiple mono datasets for use in a composite, make sure to use one set's finished stack (preferably the one with the strongest signal) as a reference to stack the others with.
• If possible, set your stacker to output 32-bit integer FITS files.
  

Research your camera, sensor and mount and familiarise yourself with any quirks of your setup. Some common quirks to be aware of and mitigate;

• Avoid lossy RAW compression where possible. If not possible, concentric rings may form in your images and calibration frames (Nikon D series), or small stars may be completely filtered out ('star eater' problem, on some Nikon and Sony models).
• Find and implement unity gain for your OSC or mono CCD (unless your circumstances or specific object require a higher gain)
• Use low-speed transfer ('download') of frames, which may avoid increased noise on some models (e.g. QHY series).
• Establish the time it takes for vibrations to settle in your setup when dithering between frames; implement a suitable pause between frames.
• Use an IR/UV cut filter (aka 'luminance filter') if using an instrument that has sensitivity past the visual spectrum, if you wish to capture visual spectrum coloring, see the Color module for details.
  

Some common problems in StarTools, caused by ignoring the check-lists above;

• Achieving results that are not significantly better than from other software
• Trouble getting any coloring
• Trouble getting expected coloring
  
• Trouble getting a good global stretch
• Halos around dust specks, dead pixels or stacking artifacts
• Finding 'nebulosity' that is not real
  
• Faint streaks (walking noise)
  
• Vertical banding
• Noise reduction or other modules do not work, or require extreme values to do anything
  
• Ringing artifacts around stars
• Color artifacts in highlights (such as star cores)
• Trouble replicating workflows as seen in tutorials and/or videos
  

Allowed / not allowed

Allowed in your dataset;

• Uncorrelated noise grain (e.g. noise grain should be exactly one pixel in size)
  
• Light pollution
• Sky gradients

Ideally avoided;

• Vignetting
• Gradients due to uneven lighting
  
• Dust specks, dust donuts
• Smudges
• Amp glow
  
• Dead pixels, dead sensor columns
  
• Satellite trails
  
• Trees or buildings
• Banding
  
• Walking noise and other correlated noise (e.g. noise that is not single-pixel speckles)

The above are all easily avoided by good acquisition techniques, correct stacker settings, and proper calibration with flats and - optionally - darks and/or bias frames.

Do's and don'ts

Do;

• Process your dataset from start-to-finish in StarTools including compositing (LRGB, LLRGB, SHO, HOO, etc.)
  
• Use simple workflows and familiarize yourself with the 'standard' suggested workflow outlined in the application itself, the many tutorials, the documentation and as roughly depicted in the home screen when reading the modules left-to-right, top-to-bottom.
  
• Acquire and apply flats
  
• Dither between frames during acquisition as often as practical (ideally every frame)
  
• Bin your dataset if your dataset is oversampled
• Use deconvolution to restore detail if possible
• Use an outlier rejection algorithm in your stacker (Median if < ~20 frames, any other more sophisticated outlier rejection algorithm if more)
  
• Practice with some publicly available datasets that are of reasonable quality to get a feel for what a module is trying to do under normal circumstances
• Align all channels/bands during stacking by using one stack as a reference for stacking the others.
  

Don'ts;

• Do not post-process any part of your image in any way, in other application before opening it in StarTools
• Do not make composites in any other application but StarTools
  
• Do not process any part of your subs in any way, in other application before stacking them
  
• Do not visit the same modules many times
• Do not process your dataset at higher resolution than necessary
• Do not drizzle your dataset in your stacker if your dataset is already oversampled
• Do not try to hide issues by clipping the interstellar background to black (this is hard to do in StarTools as it is very bad practice, but is not impossible)
• Do not mix different frames shot with exposure times or ISOs in your stacker.
• Do not align finished stacks after stacking.