![]() Remember that bit about not having dev resemble prod? We also didn't have end to end testing systems for a very large part (the only one I was aware of was the nuclear fallout calculator, whose testing was rotated around the host countries weather agencies every month). If there wasn't a record breaking cyclone that day, I doubt we would have solved the problem in 5 minutes 4 months down the line. Took 5 minutes to grab the details from yesterday's dump and rollback, and then the model's outputs flowed again. Ah climate change, you've fucked us again.īut when the model failed to get its outputs to the downstream systems, yesterday's change to the firewall was fresh in my mind. The next day, a tropical cyclone spawned in our region - an unheard of thing for July 1 - they don't usually start up til November or so. One day, in the middle of the dry season (Jun 30), I was doing the final step in a cutover to a new system - disabling the firewall rules for the old. But also, not having any kind of dev that at all resembled prod. Shitload of infrastructure and dozens to hundreds of people behind each one, not something that could simply be resurrected by git pulling and pushing to some new location in a disaster. ![]() Same code as what calculated the city models, the state models, the regional model and the global model, just very very different initial and boundary conditions. Specifically, the national weather bureau, and applications like a zoomed in mobile model centred on a tropical cyclone (or equally, the program to calculate the propagation of tsunamis). ![]() Years ago, I worked in a field where random applications would be rarely used, but it was very important that they ran when the need to run them ad-hoc came up. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |