{"id":518,"date":"2009-12-16T23:22:27","date_gmt":"2009-12-16T23:22:27","guid":{"rendered":"http:\/\/192.168.10.99\/KES\/?p=518"},"modified":"2016-09-26T15:48:30","modified_gmt":"2016-09-26T15:48:30","slug":"518","status":"publish","type":"post","link":"https:\/\/www.kandrsmith.org\/KES\/index.php\/2009\/12\/16\/518\/","title":{"rendered":"What Happened on Impact Night?"},"content":{"rendered":"<p>From the LCROSS Flight Director (Paul Tompkins) Blog. Reposted from\u00a0<a href=\"https:\/\/blogs.nasa.gov\/lcrossfdblog\/2009\/12\/16\/post_1260990722929\/\">https:\/\/blogs.nasa.gov\/lcrossfdblog\/2009\/12\/16\/post_1260990722929\/<\/a><\/p>\n<p><em>Kimberly Ennico Note: I contributed the information in Mark Shirley\u2019s addition to this entry in Paul Tompkins (Mission Operations Manager) LCROSS blog series about what was happening on the LCROSS payload during impact night. Mark was sitting in the MOCR\/Mission Operations Control Room while I was in the SOC.\/Science Operations Control across the hallway.<\/em><\/p>\n<p>The LCROSS mission was ultimately focused on the final four minutes of flight, starting at the time of the Centaur impact, and ending with the impact of the Shepherding Spacecraft.\u00a0 During that time, the Payload Engineer and the Science Team took operational center stage.\u00a0 Once the science payload was powered on, the team\u2019s job was to confirm the full functionality of the instruments, and then to adjust instrument settings to make sure the data we received was the best it could possibly be.\u00a0 For Impact, there were no second chances \u2013 the Shepherding Spacecraft was to be destroyed as a forgone outcome of its observation of the Centaur lunar collision.<\/p>\n<p>In this post, I\u2019ve invited Payload Engineer Mark Shirley to provide his perspective of impact night.\u00a0 Mark was the Payload Software Lead, in charge of the design and implementation of onboard instrument command sequences and other onboard software components, as well as a large fraction of the science-related data processing software used on the ground.\u00a0 During flight, Mark sat in the Mission Operations Control Room (MOCR) with the rest of the Flight Operations team.\u00a0 On the final day, his job was to assess and maintain the engineering functionality of the science payload in those minutes before impact.<\/p>\n<p>In the impact video sequence, from the public\u2019s perspective, there were a lot of things that happened operationally that were probably difficult to understand. \u00a0Mark will explain the plan for gathering data from the impact and also describe what actually happened in flight.\u00a0 On a lighter note, Mark was one half of our now famous (or notorious) \u201chigh-five malfunction\u201d that created such a buzz on the social media circuit after impact.\u00a0 He\u2019ll explain that as well. I\u2019ll let Mark take it from here.\u00a0 Enjoy!<\/p>\n<p>I\u2019d like to describe our plan for collecting data about the Centaur impact from the shepherding spacecraft (S-S\/C), how actual events differed from the plan, and what that says about the process of developing and flying spacecraft.\u00a0 In particular, I\u2019ll cover why some of the pictures were fuzzy and some were white and why we were sending commands during the last minutes.\u00a0 I won\u2019t touch on the scientific interpretation of the data, only the process of gathering it.\u00a0 This story contains some hard work, a few mistakes, a little nail biting tension, and finally, success.<\/p>\n<p>Central to the story is the type of mission LCROSS was: a cost-capped, fixed-schedule mission.\u00a0 That meant if LCROSS had been late, LRO would have flown with a dead or inactive LCROSS.\u00a0 If the project had run out of money, whatever hadn\u2019t been done wouldn\u2019t have gotten done.\u00a0 Within the LCROSS project, the instruments were in a similar position.\u00a0 The original idea was to observe the Centaur impact from Earth only.\u00a0 Onboard instruments were soon added to the design but with a total instrument budget of approximately $2 million.\u00a0 That\u2019s much less than single instruments on many other missions.<\/p>\n<p>Two things made success possible.\u00a0 First, project managers kept a tight focus on using the barest minimum hardware and testing required to perform the science, and went beyond that, only as the budget allowed, to increase the likelihood of success.\u00a0 Second, everyone stayed on budget. The payload team had no choice, but if any other part of the project had overrun by a lot, the payload might have been eliminated or flown only partially ready.<\/p>\n<p><strong>The LCROSS Instruments<\/strong><\/p>\n<p>LCROSS carried nine instruments.\u00a0 Five were cameras to take pictures over a large range of wavelengths, that is, colors. One was for visible light that our eyes can see.\u00a0 Two near-infrared cameras captured mineralogy and water signatures, and two mid-infrared cameras captured thermal signatures from -60C to +500C).\u00a0 LCROSS carried three spectrometers to measure color very precisely.\u00a0 One covered ultra-violet and visible wavelengths and two covered near-infrared wavelengths.\u00a0 The latter spectrometers were the best at searching for water, and one looked down toward the Centaur impact the vapor cloud it kicked up and the other looked to the side as LCROSS passed through that cloud.\u00a0 Finally, LCROSS carried a high-speed photometer to measure the brightness of the impact flash.<\/p>\n<p>The instruments are described in detail <a href=\"http:\/\/www.nasa.gov\/mission_pages\/LCROSS\/spacecraft\/instruments1.html\">here<\/a>.\u00a0 Data from all nine instruments had to share LCROSS\u2019 one Mbps (one megabit or million bits per second) radio link to the ground.\u00a0 At that rate, it takes 2 seconds to transmit a typical cell phone picture.\u00a0 This was the maximum data rate available for the LCROSS mission and used only twice: lunar swingby on June 22<sup>nd<\/sup> and impact on October 9<sup>th<\/sup>. All other instrument activities that took place during the 112 day mission used speeds less than 256 kbps (kilobits or thousand bits per second), which was sufficient for collecting data to calibrate the instruments.<\/p>\n<p><a href=\"http:\/\/www.nasa.gov\/mission_pages\/LCROSS\/spacecraft\/instruments1.html\">Link to LCROSS Instruments<\/a><\/p>\n<p><strong>The Observation Plan<\/strong><\/p>\n<p>The two components of the LCROSS mission, the Centaur and the Shepherding Spacecraft (S-S\/C), separated about 10 hours before they reached the moon.\u00a0 At the moment the Centaur impacted, the S-S\/C was still 600 kilometers above the surface.\u00a0 Falling at 2.5 kilometers per second, the S-S\/C reached the surface 4 minutes later.\u00a0 Observations of the Centaur impact event made during those 4 minutes were the purpose of the mission.\u00a0 Unlike orbital missions that can usually try multiple times to collect data, we had just one shot.<\/p>\n<p>The diagram below shows the plan for observing from the S-S\/C, starting one minute before Centaur impact, at the beginning of what we called \u201cSequence 2\u201d in the NASA TV video.\u00a0 The diagram plots our intended schedule of instrument observations against time: each row represents one of the instruments (instrument abbreviations appear below each row of data), and each tick mark along a row represents one observation, either an image or a spectrum.\u00a0 Over some intervals, the observations are spaced so closely that the plot looks like a solid bar.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009194main_ShirleyBlog_ObservationPlanCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-523\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009194main_ShirleyBlog_ObservationPlanCropped.bmp\" alt=\"1009194main_shirleyblog_observationplancropped\" width=\"914\" height=\"482\" \/><\/a><\/p>\n<p>Figure 1 The LCROSS impact observation plan: These timelines indicate when each image and spectra was planned to occur during the final four minutes of the mission.\u00a0 The horizontal axis represents time.\u00a0 Each row represents an instrument, and each tick mark represents the timing of a sample (an image or spectrum) from that instrument.<\/p>\n<p>The last four minutes were divided into three periods, called FLASH, CURTAIN and CRATER.\u00a0 Each period focused on a different aspect of the expected impact event and emphasized data collection from different instruments.<\/p>\n<p>FLASH started one minute before the Centaur impact and focused on the very short burst of light generated by the Centaur impact itself.\u00a0 Starting from the top of the diagram, the plan was to stop both the Visible Light Camera (VIS) and the Near-Infrared Camera #2 (NIR2).\u00a0 This would allow us to focus on NIR1 images which we felt had the best chance of catching the location of the impact flash which we expected to be visible for less than one second.\u00a0 These three cameras shared a common input to our payload computer, called the Data Handling Unit (DHU), and could not be used simultaneously. By stopping VIS and NIR2, we could run NIR1 at a faster rate (see the segment labeled \u2018A\u2019), increasing the odds it would image the flash.\u00a0 The planned sequence also increased the NIR1 exposure time to capture the flash signature even if it was very faint.\u00a0 We knew this would produce a badly overexposed image of the illuminated lunar surface, but our goal was to locate the impact.\u00a0 We\u2019d have plenty of other pictures of the surface.\u00a0 This sort of shifting attention between cameras accounts for the periods where one camera image would stop updating for a while.<\/p>\n<p>We also designed the FLASH strategy for the spectrometers around our expectation of a dim, short duration flash event.\u00a0 Near Infrared Spectrometer #1 (NSP1), the main water-detection instrument, was put into a high-speed, low resolution mode (represented by the yellow bar).\u00a0 The Visible and Ultraviolet light Spectrometer (VSP) was commanded to take long exposures, and Total Luminescence Photometer (TLP) was powered early enough to reach equilibrium and be at its most sensitive for the flash event.<\/p>\n<p>The second phase, CURTAIN, started just after the Centaur impact and ran for three minutes.\u00a0 Its purpose was to take spectra and images of the expanding vapor and dust clouds thrown up by the impact.\u00a0 CURTAIN was the most important period and also the simplest.\u00a0 All instruments ran in their default modes, as follows.\u00a0 The DHU shifted between the three analog cameras in a stuttering pattern \u2013 VIS, VIS, NIR1, NIR2 \u2013 repeating.\u00a0 Both thermal cameras monitored the plume shape and temperature. The two downward-facing spectrometers (NSP1 and VSP) looked for water and other chemicals.\u00a0 The side-looking spectrometer (NSP2) also looked for water and other compounds, but from sunlight scattered or absorbed by the dust and vapor cloud.\u00a0 The TLP continued to take data during this period, but it\u2019s primarily function was during FLASH.<\/p>\n<p>The goal of CRATER, the final period, was to image the crater made by the Centaur impact to get its precise location and, more importantly, its size.\u00a0 From its size and their detailed models of crater formation, the LCROSS Science Team can potentially tell us how the crater evolved over the few seconds of its formation and how much material was excavated.\u00a0 The primary instruments in this period were the two thermal cameras, MIR1 and MIR2.\u00a0 Their sample rates were increased relative to those for CURTAIN.\u00a0 To image the crater in a second frequency band, NIR2, the more sensitive near infrared camera, was commanded to its most sensitive setting.\u00a0 NIR1 and VIS would not be used during this period because neither was sensitive enough to see anything in the permanently shadowed area.\u00a0 All spectrometers would continue running to look for light reflected off of any plume or vapor cloud.\u00a0 At the end of this phase, the S-S\/C would fall below the rim of Cabeus Crater, cutting off radio transmission to Earth, and then impact the surface a couple of seconds later.<\/p>\n<p>There were three keys to making this plan work:<\/p>\n<ul>\n<li>Downlink Bandwidth: the data collected had to fit within the 1 megabit radio downlink. We did a lot of testing before launch to work out a data collection plan that was further confirmed and refined based on on-orbit performance. We gave priority to data from the most important instrument, the near-infrared spectrometers, to provide robustness to the design. The best scheme was pre-programmed and ready to go in case we were unable to command the spacecraft in the final hour.<\/li>\n<li>Camera Exposure: We had to change camera exposure settings during the descent to reflect the changing\u00a0\u00a0\u00a0 brightness of the impact event and the surrounding scene.\u00a0 Defaults were pre-programmed based on the latest lighting models for impact morning from NASA Goddard Space Flight Center.<\/li>\n<li>Command Timing: In the instrument command sequences governing FLASH, CURTAIN and CRATER periods, we had to orchestrate changes in instrument configuration as they were needed to focus on\u00a0\u00a0\u00a0 different aspects of the impact event.\u00a0 Sometimes these changes had to be interleaved with instrument data\u00a0\u00a0\u00a0 collection in a way that was vulnerable to small timing changes.<\/li>\n<\/ul>\n<p><strong>So What Actually Happened?<\/strong><\/p>\n<p>Well, as reflected in the recent LCROSS press briefing, we collected a very rich and interesting data set that met the needs of our science objectives.\u00a0 However, we had challenges in all three areas \u2013 bandwidth, exposure settings and timing \u2013 although all ultimately proved minor. However, in some ways, it was a close call.\u00a0 This diagram shows what data was actually collected during the final four minutes of the mission.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009195main_ShirleyBlog_ObservationActualCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-524\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009195main_ShirleyBlog_ObservationActualCropped.bmp\" alt=\"1009195main_shirleyblog_observationactualcropped\" width=\"922\" height=\"500\" \/><\/a><\/p>\n<p>Figure 2 Actual performance of the LCROSS payload: These timelines indicate when the images and spectra were taken on the morning of the impact.\u00a0 The pattern gives clues about the performance of the hardware \u00a0and software systems that collected the data.<\/p>\n<p>First, the rows representing the spectrometers, NSP1, NSP2 and VSP, look almost exactly as they should.\u00a0 Except for one problem with the Visible and Ultraviolet light spectrometer (VSP), which I describe later, our plan for collecting spectra worked perfectly.\u00a0 This is very good, because the spectra carried most of the information we were trying to collect.<\/p>\n<p>As for the cameras, several differences from the plan jump out.\u00a0 The most obvious is that the timing of observations along some timelines is irregular with many observations missing, e.g., the visible camera pointed to by note E. This occurred with all five cameras (the first five timelines) but not with the spectrometers (the next three timelines).<\/p>\n<p><strong>Scene Complexity and Bandwidth Limitations<\/strong><\/p>\n<p>Image compression is the process of finding and reducing redundancy in an image in order to transmit it more efficiently.\u00a0 On LCROSS, to fit within the 1 Mbps data rate limitation for our downlink, we used a lossy compression algorithm that typically reduced each image to 1\/20th its original size.\u00a0 Lossy methods achieve greater compression than lossless methods by actually removing parts of each image.\u00a0 The algorithm tries to find subtle details whose removal the human eye won\u2019t notice.\u00a0 Being able to combine many different kinds of data into a single digital data stream is so useful that this approach has been standard practice for many years.<\/p>\n<p>In flight, the irregularity of observations occurred because we underestimated the complexity of the lunar scene during ground testing.\u00a0 We had done much of our testing with a large reproduction of the moon\u2019s pole in front of the cameras, but it turned out this didn\u2019t mimic the high contrast and detail of the real scene.\u00a0 Scene complexity mattered because the images were highly compressed and changes in the moon scene changed the sizes of the compressed images by a factor of 4. We first observed this behavior during the lunar swingby LCROSS performed during the first week of its mission.\u00a0 Turning on the instruments during the swingby was intended as a learning experience, and it proved critically important.\u00a0 It provided the best operational practice we got for the impact as well as data to calibrate the instruments.<\/p>\n<p>After the lunar swingby in June, I changed the thermal camera sampling rates in the instrument command sequences for the final hour.\u00a0 Unfortunately, the compression problem turned out to be about 20% worse during the final hour of the mission than during the lunar swingby.\u00a0 This forced us to change the thermal camera rates again in real-time, but we had practiced changing them during rehearsals, just in case.\u00a0 In the NASA TV impact video sequence, you can hear the Science Team requesting a change of MIR1 rates to 1 Hz, and MIR2 to 0.1 Hz.\u00a0 See note F in the figure.\u00a0 The rate for thermal camera #1 (MIR1) changes just before this note and changes for MIR2 just after it.\u00a0 Even though we\u2019d practiced, this was still a very tense time as we were losing some data while the changes were being made. Changing the MIR rates felt like it took forever.<\/p>\n<p>The bandwidth problem could have been avoided if in addition to changing camera sampling rates in the command sequences, we had also changed the stuttering pattern for the analog cameras mentioned above to eliminate one VIS image during each repetition.\u00a0 However, our instrument simulator didn\u2019t have the full set of instruments like the spacecraft, which made it impossible to adequately test this change on the ground.\u00a0 At one point, we discussed testing this change onboard before the impact, but lost the opportunity due to the fuel loss Paul described in his blog on October 4<sup>th<\/sup> (see \u201cA Test of the Flight Team\u201d).<\/p>\n<p>One other problem caused by the complex lunar scene was damaged images.\u00a0 After compression, some of the visible camera images were still too large to fit within a single data packet for transmission to Earth.\u00a0 Here\u2019s an example of the kind of damaged image that resulted.\u00a0 The shadowed area should be completely dark, but instead contains wispy bright areas. These compression-artifacts are intimately linked to the scene and need to be taken out with image post-processing.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009196main_ShirleyBlog_CraterImageArtifactsCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-525\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009196main_ShirleyBlog_CraterImageArtifactsCropped.bmp\" alt=\"1009196main_shirleyblog_craterimageartifactscropped\" width=\"531\" height=\"528\" \/><\/a><\/p>\n<p>Figure 3 Example of damage to downlinked images due to clipping in the telemetry packet formatting software.<\/p>\n<p>What caused these compression artifacts?\u00a0 I didn\u2019t know it at the time, but the software for compressing these images had been written some years before to clip the compressed form of images to ensure they always fit within a single data packet (maximum size 65536 bytes).\u00a0 We used a wavelet-based compression algorithm, and clipping the compressed images removed some information needed to recreate the image accurately.\u00a0 The alternative would have been to split the images across multiple packets and reassemble them on the ground.\u00a0 This certainly could have been done in principle, but doing so would have introduced significant changes right at the heart of software that we had planned to reuse without change after its successful use on previous projects.\u00a0 With what we know now, changing this would have been justified, but it would also have been risky given how central image compression was to the overall design.\u00a0 With what we knew at the time, I believe we would have left it alone because of our short development schedule and all the other things that had to work.<\/p>\n<p><strong>November, November!<\/strong><\/p>\n<p>Most of the commanding we did from the ground was to adjust the exposure times of the near infrared cameras as the scene changed.\u00a0 The other cameras either controlled themselves (VIS) or had only one appropriate setting (the thermal cameras, MIR1 and MIR2).\u00a0 We controlled the exposure setting for the near infrared cameras explicitly because we were trying to image a relatively dim flash and ejecta curtain close to bright mountain peaks.<\/p>\n<p>Near the beginning of the FLASH period, we discovered we didn\u2019t get this balance right. To image the dim centaur impact flash, we deliberately overexposed the sunlit peaks.\u00a0 This setting combined with the Cabeus scene overdid it. The sunlit areas electronically bled into nearby parts of the image.\u00a0 That occurs when electrons in overexposed pixels move across the image detector to other pixels.\u00a0 In this case, the shadowed area of Cabeus crater was completely covered, obscuring our view of the impact.\u00a0 That was why the only image that was updating just before the Centaur impact was white.\u00a0 We hadn\u2019t seen this level of bleeding earlier in the mission, or in almost any of our testing.\u00a0 However, after searching through our data archive, I realize now we did see it occur once, two years ago, in one flashlight test in a darkened room but did not fully comprehend the implications.<\/p>\n<p>The FLASH period was designed to start 1 minute before the Centaur impact, so we had a little time to recover once we saw the problem.\u00a0 During this minute, our first priority was to confirm that the spectrometers (NSP1 and VSP) and photometer (TLP) were working properly.\u00a0 Once that was done, we focused on the NIR1. Since we still had commanding during this period, we tried to change the exposure setting (Payload Scientist Kim Ennico called out \u201cFlight, this is Science, please change NIR1 to OPR 9, over.\u201d). We had less than 30 seconds to get this command sent up to the spacecraft. \u00a0The command was actually sent but arrived a few seconds too late to capture the impact. In hindsight, this was a challenging stretch for the camera\u2019s range, due to the scene and the potential for bleeding. Our strategy \u2013 to aim for the most sensitive exposure setting followed by one attempt to back-off depending on the data-might have worked had we been looking at another region of the moon, that is, had, we launched (and impacted) on a different date, where the terrain and lighting would have been different. While all this was going on, the impact flash was captured by NSP1, so the key science measurement was made.<\/p>\n<p>We intentionally caused the same issue later, during the CRATER period, but we had better success (see above figure at the segment labeled \u2018B\u2019).\u00a0 Initially, the NIR2 camera images were badly overexposed for the same reason as during FLASH (hence the white images that appear in the NASA TV video just after entry to DV Mode).\u00a0 Kim Ennico, the Payload Scientist, made the call to reduce the exposure time slightly, from what we called OPR 15 to OPR 10. \u00a0(Again, you can hear this request over the voice loop in the NASA TV video.)\u00a0 She was using live-information from the NSP1 spectrometer and checking those values in real time against a spreadsheet near her seat.\u00a0 You can see her checking and rechecking on the video before making the choice.\u00a0 We only had only one chance to choose the right one.\u00a0 The command was sent and received 30 seconds before the S-S\/C\u2019s impact.\u00a0 Kim\u2019s call initially left the images overexposed, but as the lit peaks slid out of the field of view, her choice produced excellent images of the very dark crater floor, including the image that gave us our best estimate of the Centaur crater size.\u00a0 These images go all the way down to 2 seconds before S-S\/C impact where the craft was 5 kilometers above the surface. The crater floor of Cabeus was indeed brighter than any of the predictions, at least in the infrared. That\u2019s another reminder of science and exploration. Sometimes you are surprised as you collect new data, especially data from areas never looked at before.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009197main_ShirleyBlog_Saturated-ImageTrioCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-526\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009197main_ShirleyBlog_Saturated-ImageTrioCropped.bmp\" alt=\"1009197main_shirleyblog_saturated-imagetriocropped\" width=\"927\" height=\"223\" \/><\/a><\/p>\n<p>Figure 4 This image sequence was captured just before the end of the mission and shows the NIR2 camera going from badly overexposed to acceptably exposed as the lit peaks surrounding Cabeus leave the field of view.<\/p>\n<p>How do we know that\u2019s the Centaur crater?\u00a0 Because it was also seen by the two thermal cameras (MIR1 and MIR2) while it was still warm, and we can overlay the images.\u00a0 The left figure below shows aligned images from NIR2 and MIR1, taken before the Centaur impact.\u00a0 The figure on the right shows aligned images from these cameras taken just before the S-S\/C impacted and showing the Centaur impact crater (see inset).\u00a0 These images don\u2019t align perfectly because they were taken about a second and 2.5 kilometers apart.\u00a0 To obtain images of the Centaur crater using three different cameras was our goal, and we succeeded.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009199main_ShirleyBlog_OverlayedCraterCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone  wp-image-527\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009199main_ShirleyBlog_OverlayedCraterCropped.bmp\" alt=\"1009199main_shirleyblog_overlayedcratercropped\" width=\"411\" height=\"308\" \/><\/a><\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009200main_ShirleyBlog_InsetCentaurCraterCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone  wp-image-528\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009200main_ShirleyBlog_InsetCentaurCraterCropped.bmp\" alt=\"1009200main_shirleyblog_insetcentaurcratercropped\" width=\"406\" height=\"307\" \/><\/a><\/p>\n<p>Figure 5 The bottom image shows the Centaur impact crater in both near-infrared and mid-infrared images.\u00a0 The top image overlays images taken before the impact by the same two cameras.<\/p>\n<p>Kim\u2019s call of \u201cNIR2 to OPR 10\u2033 yielded a great image of the Centaur crater, but it also caused some confusion.\u00a0 The name for Near-Infrared Camera #2 (NIR2) was too similar to the name of Mid-Infrared Camera #2 (MIR2).\u00a0 We had practiced this interaction over our voice communication loops, but we hadn\u2019t practiced it enough to do it quickly and perfectly under time pressure.\u00a0 Kim had to repeat her call using the phonetic term \u2018November\u2019 for the \u2018N\u2019 at the beginning of NIR2 (during CURTAIN phase in the NASA TV video, you can hear the Flight Controller, Jim Strong, ask \u201cis that \u2018November\u2019 or \u2018Mike\u2019?\u201d, referring to NIR2 or MIR2, ).\u00a0 We didn\u2019t realize when we picked the obvious names for these cameras three years ago that the names could cause confusion when spoken over the voice loops connecting our mission control rooms.\u00a0 Back then, our plan avoided real-time commanding completely, but as we learned through our practice sessions before launch and actual experience after launch, we realized we needed the flexibility.\u00a0 We retrofitted a process for proposing and confirming real-time commands into our mission operations architecture as best we could given the facility and time constraints.\u00a0 Note that we didn\u2019t consider doing this for maneuvering the spacecraft or for the most critical science instruments, only the secondary instruments.<\/p>\n<p><strong>Command Timing<\/strong><\/p>\n<p>The commanding side of the automatic sequence ran almost perfectly.\u00a0 We did have one problem with the Visible and Ultraviolet Spectrometer during the CURTAIN period, though.\u00a0 Because the instrument data handling unit (DHU) was at its maximum data throughput capacity during the first part of CURTAIN, one command to change exposure time was delayed and sent during a period when the instrument wasn\u2019t listening.\u00a0 That command was ignored.\u00a0 This resulted in capturing fewer spectra with longer-than-planned exposure times.\u00a0 Luckily, the longer exposure times turned out to be a blessing, since the ejecta curtain was much fainter than some models predicted. The loss of more frequent sampling due to the longer exposures did not affect the science measurement.<\/p>\n<p><strong>High Five Fail<\/strong><\/p>\n<p>Yes, I should have.\u00a0 After the end of the mission, I missed a high five that was captured on camera.\u00a0 I was teased about it by my colleagues that morning and by my kids that night. The other operator involved, our Telemetry Data Manager, known as \u201cData\u201d over the voice loop, is both a good colleague and a friend.\u00a0 I didn\u2019t intend to embarrass him and have since apologized.\u00a0 We had been told to avoid high fives to prevent exactly the sort of mistake I made, but once the hand went up, I should have responded.\u00a0 So, here it is, for the record:<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009201main_ShirleyBlog_HiFive.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-529\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009201main_ShirleyBlog_HiFive.bmp\" alt=\"1009201main_shirleyblog_hifive\" width=\"327\" height=\"245\" \/><\/a><\/p>\n<p>Why didn\u2019t I respond?\u00a0 I honestly don\u2019t remember the moment clearly, but I did have two things on my mind.\u00a0 First, my job at that point was to move to the Science Operations Center (SOC) to prepare for the post-impact press conference.\u00a0 We had two hours to make sure we\u2019d gotten the data we expected, to prepare presentation charts and to look for anything obvious.\u00a0 I was concerned about that because I had missed seeing the plume we had hoped for like most everyone else.<\/p>\n<p>More importantly, I was really stressed out.\u00a0 The DHU, the computer that processed all instrument data, was struggling with the very large packet sizes of visible camera images, and the DHU almost crashed a number of times during that final hour.\u00a0 We had developed and practiced a procedure to recover from such a crash to prevent a substantial loss of science data. \u00a0Flying a spacecraft is a group effort with lots of cross-checking, and as the Payload Software Lead, I felt especially responsible.<\/p>\n<p>During payload development testing, we found and fixed several problems that would have been problematic for the payload.\u00a0 This problem which led to potential crashes of the DHU was known and was the most difficult software problem we saw.\u00a0 The root of the problem was a small chip that controlled the data bus connecting the video capture and compression chips to each other and to the main processor within the DHU.\u00a0 Under certain circumstances this bus controller chip would stop responding, and the DHU software would crash.\u00a0 Since we didn\u2019t have access to the chip\u2019s design to understand why it would stop, and we didn\u2019t have time to replace it, our approach was to create a method for quickly recovering on orbit.\u00a0 This method had two parts.\u00a0 The first part was a software patch we developed that reset the bus controller when the DHU\u2019s main processor noticed it had stopped responding.\u00a0 The second part was a procedure for quickly rebooting the whole DHU from the ground if the software patch didn\u2019t catch the problem.<\/p>\n<p>We developed and tested the software patch just a few weeks before the payload was shipped to Southern California for integration with the rest of the spacecraft. From that point on, through the rest of our testing on the ground and in orbit, we didn\u2019t see this problem again. That is, we didn\u2019t see it until the morning of the impact.\u00a0 That morning, the patch needed to reset the bus controller two dozen times.\u00a0 The vertical green lines in this figure show when.<\/p>\n<p><a href=\"\/KES\/wp-content\/uploads\/2009\/12\/1009202main_ShirleyBlog_DHURebootActualCropped.bmp\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-530\" src=\"\/KES\/wp-content\/uploads\/2009\/12\/1009202main_ShirleyBlog_DHURebootActualCropped.bmp\" alt=\"1009202main_shirleyblog_dhurebootactualcropped\" width=\"958\" height=\"356\" \/><\/a><\/p>\n<p>Figure 6 During the final hour of the mission, Data Handling Unit (DHU) software detected and corrected an anomalous condition on a bus controller chip multiple times.\u00a0 The green lines show when these events occurred.\u00a0 The right end of this figure, starting at the label \u2018I\u2019 (the time of Centaur impact) corresponds to the time spans of the planned (Figure 1) and actual (Figure 2) performance plots. Earlier events happened while monitoring payload performance in the 56 minutes prior to Centaur impact.<\/p>\n<p>Once these events started, I was prepared, on a hair trigger, to start the process of rebooting the DHU if the patch didn\u2019t work.\u00a0 I was constantly checking and rechecking the fault response procedure I had developed for our payload.\u00a0 The details of this procedure varied over time.\u00a0 As the on-board sequence progressed and we got closer and closer to the Centaur impact, we had different decisions to make to recover if something went wrong.\u00a0 This strategizing was being done over another voice loop with Kim and Tony Colaprete, the LCROSS Principal Investigator, in the Science Operations Center (SOC) which was not audible to the audience watching on NASA TV.\u00a0 We had to keep track of a lot of independent data threads and contingencies simultaneously.\u00a0 Our actual trigger for starting this recovery, a gap in numeric sequence of the data coming from the instruments, even occurred once, but it was unrelated and didn\u2019t need a response.\u00a0 In the end, our defenses worked.\u00a0 The software patch performed exactly as intended and no crash occurred.\u00a0 After the S-S\/C impact, I breathed a sigh of relief and moved to the next room to start preparing for the press conference.<\/p>\n<p><strong>Final Thoughts<\/strong><\/p>\n<p>As I said above, we had challenges in all three areas critical to making our plan work: downlink bandwidth, camera exposure, and command timing.\u00a0 Ultimately, all of the problems we had proved minor, and we collected the data we needed to draw conclusions about the presence or absence of water and other substances in Cabeus crater.<\/p>\n<p>We at NASA all too often strive to give the impression that complex, difficult missions are routine.\u00a0 They\u2019re not.\u00a0 They\u2019re complex and difficult.\u00a0 What makes them possible is long planning, teamwork, and careful review by people both inside and outside the project.\u00a0 One name for this process is \u201cSystems Management\u201d, which recognizes that people need backup just like the parts of a complex machine.\u00a0 I personally made some mistakes and caught some mistakes.\u00a0 Together, we caught enough of them that we were successful.\u00a0 For me, it was a huge privilege and a wonderful experience.\u00a0 I\u2019m very grateful to have been a part of this mission.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>From the LCROSS Flight Director (Paul Tompkins) Blog. Reposted from\u00a0https:\/\/blogs.nasa.gov\/lcrossfdblog\/2009\/12\/16\/post_1260990722929\/ Kimberly Ennico Note: I contributed the information in Mark Shirley\u2019s addition to this entry in Paul Tompkins (Mission Operations Manager) LCROSS blog series about what was happening on the LCROSS payload during impact night. Mark was sitting in the MOCR\/Mission Operations Control Room while I &hellip; <a href=\"https:\/\/www.kandrsmith.org\/KES\/index.php\/2009\/12\/16\/518\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">What Happened on Impact Night?<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[11],"tags":[33,34],"class_list":["post-518","post","type-post","status-publish","format-standard","hentry","category-science","tag-lcross","tag-moon"],"_links":{"self":[{"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/posts\/518","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/comments?post=518"}],"version-history":[{"count":0,"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/posts\/518\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/media?parent=518"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/categories?post=518"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kandrsmith.org\/KES\/index.php\/wp-json\/wp\/v2\/tags?post=518"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}