The Life of a Discrepancy Investigator
Oct. 3rd, 2015 04:56 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
hrjI'd halfway promised to do this past week's Random Thursday blog on a description of my day job. In part, this was because the job was being rather intense, so it was on my mind. But the intensity was part of why I didn't (no lunch hour!) and I ended up opting to re-use to a Twitter flight instead. But now that it's Saturday, here's the promised job post.
I don't talk in a lot of specifics about my workplace because being specific would then entail spending a lot of time reminding readers that I don't in any way speak for my employer. I don't talk in specifics about the investigations I work on because somewhere in there is a fine dividing line between interesting conversation and "insider information". Very very occasionally, I'm privy to information that could affect perceptions of the company's stock value or speculative projections about upcoming business announcements. I'd rather be safe than sorry. So I'm not going to name my employer (though it would be trivially easy to figure out who it is) and I'm not going to talk about detailed specifics of my projects, just the general shapes.
I work for a biotech division of a Big Pharma company (with a capital "B" and a capital "PH"). Fortunately for my conscience, I work for a division whose product is unambiguously necessary and valuable for our patients. It's also a product where, while we may jockey against competitors for market share, there's absolutely no point in trying to "sell" it to people outside the key market. Conversely, it's an incredibly complicated drug (and will become even more complicated--in the name of becoming safer and more effective--in versions currently in development). The manufacturing process is long and involved, with a lot of potential failure points and a lot of nuances to the "SISPQ"s as we call them. (Strength, Identity, Safety, Purity, and Quality).
We start with a mammalian cell line with an introduced human gene that produces the protein that forms the essence of the drug. The manufacturing process starts with mixing up solutions (cell culture medium, various chemical solutions used in processing and purification) and thawing out a cryo-preserved vial from the "master cell bank" which was laid down a couple decades ago and has been sitting in a liquid nitrogen freezer ever since. The cells get scaled up until we have enough for a viable culture in a 200L culture vessel which is then split into other vessels until we have several running in parallel. The clock is ticking at this point because mammalian cells have a finite effective lifespan before their growth starts going wonky and the productivity efficiency falters.
Unlike, for example, a batch of beer (with which the cell culture process has a startlingly large amount in common), we do what's called "continuous perfusion" where we continuously feed in fresh cell culture medium and harvest off spent medium that has cellular byproduct in it. Among those byproducts is the protein we want. Pretty much all the other cell-piss (which is what it comes down to) needs to be removed in processing. We also don't want the cells themselves, and there are complex processes to avoid harvesting cells, or damaging the cells in the process (because we also have to remove any cell-bits that end up in the harvested material). And if you want to think about potential failure modes, consider a process that is designed to create the idea conditions for cellular growth…and then trying to make sure nothing gets in there to grow except the specific thing you want!
That's the first step, ok? Now we have an enormous volume of cell-piss to process. Let's say approximately 12,000 liters per day total from the several vessels. We start with a simple mechanical filtration to remove remaining cells and cell-bits (plus some other processes, but I'm doing a simplified overview). Then we concentrate it down and freeze it and put it in our -30C warehouse for a while. Why? Because first we have to make sure that the frozen concentrate meets all our quality standards before we invest in the next step of the process. This is where the discrepancy investigators come in. Hold that thought.
Once the frozen concentrate is declared A-OK, it moves on to purification. This involves putting it through a number of processes (I'll skip the details), each designed to remove a specific type of impurity: residual cell proteins, host cell DNA, inactive drug proteins, chemicals needed in the cell culture process that we don't need any more, viruses, and so forth. At the end of this purification process, we concentrate and freeze the result again and once more stash it in the -30C warehouse for a while. Why? I think you know the drill by now: because we need to make sure it meets all our quality requirements before we invest in the next step of the process. (See: discrepancy investigators. Hold that thought.)
The final stage is pretty simple and incredibly important: thaw it out and dilute it to the right concentration, fill a very precise volume into vials and freeze-dry it, package and label the vials. Make sure the final vials meet all our quality requirements (see: discrepancy investigators) and when it's all good, ship it off to the patients who trust us enough to inject the result into their circulatory system.
So what is it that I do? Any time something doesn't work the way we expect, I'm one of the people (discrepancy investigators) who figures out what happened, why, how, what the effects on product quality are, what the effects on our confidence in our process are, what additional information we might need to have confidence in our product and process, and--if necessary--what product we might need to reject because we don't have that confidence. It could be as trivial as someone leaving a blank on a document all the way up through discovering that our cells are producing a version of the protein that isn't biologically active. It's a very formal process with rigorous standards and multiple stages of review. And pretty much every investigation is different. (Because if they were the same, it would mean we hadn't fixed the problem the first time.)
So what was I doing this week? Well, one thing I was doing was participating in rehearsals for presentations to a national regulatory agency who will be inspecting our facilities and process next week to determine if they're ready to approve our "next generation" product. My part is to know everything there is to know about certain key investigations (whether I was the one who performed them or not) and be ready to explain and defend the process and the results.
Another thing I was doing (and this was the one that involved 5 hours of meetings every day) was helping brainstorm and write up a Risk Assessment, which is a document examining a particular "failure mode" and looking at all the possible consequences it could have and listing all the things we have in place (or could put in place) to manage or eliminate those consequences. Sometimes risk assessments are covering the normal production processes and explaining why they're properly designed. In this case, we're looking at an event where we didn't follow those processes correctly and trying to determine what the potential consequences of that mistake are.
In addition to these special projects, there's the normal daily routine: process new reports of events, triage them in terms of potential impact (most are fairly obviously trivial), assign them out to other investigators. For my own investigations, pull in production data, reports from Subject Matter Experts, personnel interviews, etc. in order to determine causes and consequences. Discuss failures with their process owners to identify corrective actions. Write everything up in reports and submit them for QA review. And just for fun, I sometimes get to do things like provide technical writing expertise (not because it's part of my basic job, but because I've been identified as being really good at it), or helping re-design a process, or pulling together historic data for a long-term review. I've ended up being one of those people who get e-mailed with questions like, "I remember something vaguely like this happening maybe five years ago. Can you dig it up?" And my favorites are things like when a director or VP drops by with a copy of a memo and asks, "I need the exact right wording to convey a very specific nuance. See what you can do with this." (Again, not because it's my job, but because I have a reputation for being good at it.)
One of the things I love about my job is that I need to know everything about everything. (Or at least: a basic familiarity about everything.) I need to know how all our equipment works from the HVAC system to the water distillation equipment to the laboratory equipment to the various instrument systems. I need to know the biological systems involved, not just for the cell culture but for microbial control. I need to understand enough about protein chemistry that I can interpret the quality data and have a sense of what possible causes to pursue when something goes wrong. Or to know when the apparent problem is so unlikely that it's probably masking something else I should be looking at. I need to know a lot of basic chemistry and physics. I need to understand all our information systems and how they inter-relate. (What's this? We received a shipment of a raw material and the computer spit out a list of required quality tests for it that we no longer perform? What the heck happened?) I need to have a keenly developed "smell test" and know when to say, "There's something non-obvious going on here and we don't look more deeply into it now we'll wish we had later." And on top of everything else, I need to understand how human beings interact with all those systems and to be able to identify when a system has been inadvertently designed to encourage the humans to fail.
Another part of "knowing everything about everything" is that we investigators regularly get shared around to balance workload. I'm officially assigned to a specific department (purification) for one specific product variant. But over the last decade I've done investigations for pretty much every stage and aspect of our process, either as a long-term assignment or as part of an ad hoc strike force. I'm very proud of the fact that my name has a habit of being brought up when there's a really tricky or really important investigation. It's not just a matter of job security (I'm not too worried about that at this point) but of pride in my work. And that's why you may sometimes hear me grousing about the intensity of the work, or the occasional long hours, but when I say, "I love my job" it is never never sarcastic.
I don't talk in a lot of specifics about my workplace because being specific would then entail spending a lot of time reminding readers that I don't in any way speak for my employer. I don't talk in specifics about the investigations I work on because somewhere in there is a fine dividing line between interesting conversation and "insider information". Very very occasionally, I'm privy to information that could affect perceptions of the company's stock value or speculative projections about upcoming business announcements. I'd rather be safe than sorry. So I'm not going to name my employer (though it would be trivially easy to figure out who it is) and I'm not going to talk about detailed specifics of my projects, just the general shapes.
I work for a biotech division of a Big Pharma company (with a capital "B" and a capital "PH"). Fortunately for my conscience, I work for a division whose product is unambiguously necessary and valuable for our patients. It's also a product where, while we may jockey against competitors for market share, there's absolutely no point in trying to "sell" it to people outside the key market. Conversely, it's an incredibly complicated drug (and will become even more complicated--in the name of becoming safer and more effective--in versions currently in development). The manufacturing process is long and involved, with a lot of potential failure points and a lot of nuances to the "SISPQ"s as we call them. (Strength, Identity, Safety, Purity, and Quality).
We start with a mammalian cell line with an introduced human gene that produces the protein that forms the essence of the drug. The manufacturing process starts with mixing up solutions (cell culture medium, various chemical solutions used in processing and purification) and thawing out a cryo-preserved vial from the "master cell bank" which was laid down a couple decades ago and has been sitting in a liquid nitrogen freezer ever since. The cells get scaled up until we have enough for a viable culture in a 200L culture vessel which is then split into other vessels until we have several running in parallel. The clock is ticking at this point because mammalian cells have a finite effective lifespan before their growth starts going wonky and the productivity efficiency falters.
Unlike, for example, a batch of beer (with which the cell culture process has a startlingly large amount in common), we do what's called "continuous perfusion" where we continuously feed in fresh cell culture medium and harvest off spent medium that has cellular byproduct in it. Among those byproducts is the protein we want. Pretty much all the other cell-piss (which is what it comes down to) needs to be removed in processing. We also don't want the cells themselves, and there are complex processes to avoid harvesting cells, or damaging the cells in the process (because we also have to remove any cell-bits that end up in the harvested material). And if you want to think about potential failure modes, consider a process that is designed to create the idea conditions for cellular growth…and then trying to make sure nothing gets in there to grow except the specific thing you want!
That's the first step, ok? Now we have an enormous volume of cell-piss to process. Let's say approximately 12,000 liters per day total from the several vessels. We start with a simple mechanical filtration to remove remaining cells and cell-bits (plus some other processes, but I'm doing a simplified overview). Then we concentrate it down and freeze it and put it in our -30C warehouse for a while. Why? Because first we have to make sure that the frozen concentrate meets all our quality standards before we invest in the next step of the process. This is where the discrepancy investigators come in. Hold that thought.
Once the frozen concentrate is declared A-OK, it moves on to purification. This involves putting it through a number of processes (I'll skip the details), each designed to remove a specific type of impurity: residual cell proteins, host cell DNA, inactive drug proteins, chemicals needed in the cell culture process that we don't need any more, viruses, and so forth. At the end of this purification process, we concentrate and freeze the result again and once more stash it in the -30C warehouse for a while. Why? I think you know the drill by now: because we need to make sure it meets all our quality requirements before we invest in the next step of the process. (See: discrepancy investigators. Hold that thought.)
The final stage is pretty simple and incredibly important: thaw it out and dilute it to the right concentration, fill a very precise volume into vials and freeze-dry it, package and label the vials. Make sure the final vials meet all our quality requirements (see: discrepancy investigators) and when it's all good, ship it off to the patients who trust us enough to inject the result into their circulatory system.
So what is it that I do? Any time something doesn't work the way we expect, I'm one of the people (discrepancy investigators) who figures out what happened, why, how, what the effects on product quality are, what the effects on our confidence in our process are, what additional information we might need to have confidence in our product and process, and--if necessary--what product we might need to reject because we don't have that confidence. It could be as trivial as someone leaving a blank on a document all the way up through discovering that our cells are producing a version of the protein that isn't biologically active. It's a very formal process with rigorous standards and multiple stages of review. And pretty much every investigation is different. (Because if they were the same, it would mean we hadn't fixed the problem the first time.)
So what was I doing this week? Well, one thing I was doing was participating in rehearsals for presentations to a national regulatory agency who will be inspecting our facilities and process next week to determine if they're ready to approve our "next generation" product. My part is to know everything there is to know about certain key investigations (whether I was the one who performed them or not) and be ready to explain and defend the process and the results.
Another thing I was doing (and this was the one that involved 5 hours of meetings every day) was helping brainstorm and write up a Risk Assessment, which is a document examining a particular "failure mode" and looking at all the possible consequences it could have and listing all the things we have in place (or could put in place) to manage or eliminate those consequences. Sometimes risk assessments are covering the normal production processes and explaining why they're properly designed. In this case, we're looking at an event where we didn't follow those processes correctly and trying to determine what the potential consequences of that mistake are.
In addition to these special projects, there's the normal daily routine: process new reports of events, triage them in terms of potential impact (most are fairly obviously trivial), assign them out to other investigators. For my own investigations, pull in production data, reports from Subject Matter Experts, personnel interviews, etc. in order to determine causes and consequences. Discuss failures with their process owners to identify corrective actions. Write everything up in reports and submit them for QA review. And just for fun, I sometimes get to do things like provide technical writing expertise (not because it's part of my basic job, but because I've been identified as being really good at it), or helping re-design a process, or pulling together historic data for a long-term review. I've ended up being one of those people who get e-mailed with questions like, "I remember something vaguely like this happening maybe five years ago. Can you dig it up?" And my favorites are things like when a director or VP drops by with a copy of a memo and asks, "I need the exact right wording to convey a very specific nuance. See what you can do with this." (Again, not because it's my job, but because I have a reputation for being good at it.)
One of the things I love about my job is that I need to know everything about everything. (Or at least: a basic familiarity about everything.) I need to know how all our equipment works from the HVAC system to the water distillation equipment to the laboratory equipment to the various instrument systems. I need to know the biological systems involved, not just for the cell culture but for microbial control. I need to understand enough about protein chemistry that I can interpret the quality data and have a sense of what possible causes to pursue when something goes wrong. Or to know when the apparent problem is so unlikely that it's probably masking something else I should be looking at. I need to know a lot of basic chemistry and physics. I need to understand all our information systems and how they inter-relate. (What's this? We received a shipment of a raw material and the computer spit out a list of required quality tests for it that we no longer perform? What the heck happened?) I need to have a keenly developed "smell test" and know when to say, "There's something non-obvious going on here and we don't look more deeply into it now we'll wish we had later." And on top of everything else, I need to understand how human beings interact with all those systems and to be able to identify when a system has been inadvertently designed to encourage the humans to fail.
Another part of "knowing everything about everything" is that we investigators regularly get shared around to balance workload. I'm officially assigned to a specific department (purification) for one specific product variant. But over the last decade I've done investigations for pretty much every stage and aspect of our process, either as a long-term assignment or as part of an ad hoc strike force. I'm very proud of the fact that my name has a habit of being brought up when there's a really tricky or really important investigation. It's not just a matter of job security (I'm not too worried about that at this point) but of pride in my work. And that's why you may sometimes hear me grousing about the intensity of the work, or the occasional long hours, but when I say, "I love my job" it is never never sarcastic.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2015-10-04 08:01 am (UTC)Definitely envy your good communication skills. I struggle with that myself!
no subject
Date: 2015-10-04 12:02 pm (UTC)no subject
Date: 2015-10-04 04:17 pm (UTC)no subject
Date: 2015-10-04 06:21 pm (UTC)I literally walked into the job having no idea what it was or that such a job even existed.
no subject
Date: 2015-10-06 08:23 am (UTC)no subject
Date: 2015-10-06 02:12 pm (UTC)I'd been working in medical sciences and biotech for a dozen years or more before I decided to do my PhD. Everything from a private medical lab (including doing blood draws) to a biotech start-up running the human tissue bank. By sheer coincidence, I'd decided to do grad school and put in my application right before the company I was working for had a set-back and laid off a quarter of the company. So instead of just quitting, I got 6 months of severance pay.
no subject
Date: 2015-10-04 07:48 pm (UTC)no subject
Date: 2015-10-05 01:18 am (UTC)no subject
Date: 2015-10-05 08:12 am (UTC)If it weren't for cell piss, I'd be dead already. Heh.