| Author: | Francesca Boffi |
|---|
ACS was involved like all other instrument teams in the INS division to test pysynphot against synphot calculations. In this effort, we proceeded step by step and were able to assess when differences (or failures) were acceptable, i.e. would be explained by the intrinsic differences in calculations of the two tools.
An important aspect was the use of diagnostic plots that were produced from the FITS files of the synphot and pysynphot spectra calculated for each test (for details see “Diagnostic plots”).
In early February 2009 testing of these was completed and ACS was able to accept all failures. The reason why we accepted them all is that they would fall clearly in one of the identified known areas of discrepancy between synphot and pysynphot. Several of our failed cases were due to the different interpolation done by the two tools (e.g. extremely narrow lines; steep gradients; differences in endpoint handling) and some to the use of incorrect spectra for the selected observation mode. In the cases where the prisms were used in the observation mode, we identified a real difference between SYNPHOT and pysynphot in the calculation of the effective wavelength. The reason for the huge difference in the calculation of this quantity was that pysynphot was calculating the effective wavelength based on the waveset specified by the wavecat file, whereas SYNPHOT was calculating it based on the native waveset.
This “discovery” was followed by a large discussion, including a discussion on the meaning of calculating an effective wavelength for a spectroscopic mode. In the end it was agreed upon by all teams that pysynphot will calculate the effective wavelength in the same way as countrate, i.e. by default this quantity will be calculated on the binned waveset from the wavecat. An interactive use may choose to calculate it on the native waveset of the observation. This is a difference from SYNPHOT.
Pinned files were needed: see “Identified data issues”.
The next step was then to run ETC 17.2.2 with SYNPHOT, first with CDBS pinned files and then with non pinned files; this comparison would allow certification of the pinned files. Finally, for the pysynphot certification we needed to compare results of the SYNPHOT+pinned run with a run pysynphot+pinned.
In the case of ACS, only two test cases failed the comparison between pinned and non pinned files. In this case we were able to assess that the difference between calculated quantities was small enough (~2%) to be ignored. All teams certified the pinned files quite straightforwardly.
Testing of ETC 17.4 (with pysynphot and pinned files)
This ETC version ran all test cases that had already been compared via the SYNPHOT/pysynphot comparison (as described in previous paragraphs). ACS got 27 failed cases, when comparing the results of the ETC 17.4 version with the ETC calculations with SYNPHOT. Again, we proceeded by reviewing all cases and assessing whether or not any could be accepted. The guiding line was to understand if the difference between calculated and expected value could be considered small enough.
All ACS failed cases were then classified into three categories, on the basis of the size of the discrepancy between expected and calculated quantities: 1) Discrep~0.01-0.013; 2) Discrep~0.019-0.02; 3) Discrep (>>0.02, between 0.5 and 0.8 !). [Discrep. is by definition equal to the difference (actual - expected) divided by the expected value].
Of all these cases, group 1) discrepancies were consistent with other teams findings; group 2) a bit higher, while group 3) indeed posed concern. What was noted by Vicki Laidler is that the group 3) “misterious or worrisome” failed cases had in some cases passed the syn_pysyn test and that most but not all involved either prisms or long pass filters, “so the difference may be coming from the fact that SYNPHOT takes the intersection, while pysynphot takes the union of the cases”.
In the end, what had been discussed about spectroscopic and imaging modes back for previous stages of testing, came to the rescue. Some of the differences can be explained by taking into account that SYNPHOT.countrate always integrates over the wavelength range of the wavecat, whereas pysynphot may integrate over the wavecat or the native waveset depending on whether the observation is imaging or spectroscopy. This was verified by having pysynphot do the same calculations by integrating over the wavelength range other than the default.
At this point, because of the very tight schedule we are on, and the need to release the ETCs in time to be used for the MCTP (Multi Cycle Treasury Programs), SSB suggested a two stage approach: 1) make ETC+Pysynphot do what the ETC+SYNPHOT does, even if it’s not entirely correct; 2) do the right thing when more time is available. This plan was embraced by all teams and a final round of testing was analyzed. These tests were aimed at comparing the results between ETC+SYNPHOT and ETC+pysynphot “a’ la SYNPHOT”. In the case of ACS all discrepancies could be understood and therefore we were able to accept all differences and all “failed” cases.