OPEN, INTEGRATED AND TAMPER-PROOF
SCADA for new production line at Bayer
describes Daniel Keck the fully automatic line at Bayer in Leverkusen, Germany. He is primarily responsible for
the SCADA system. As part of a paperless production, Bayer will fill lyophilized pharmaceuticals into vials providing
vital medication for seriously ill people. Therefore, it is not surprising that Bayer increased its requirements
for IT safety and data integrity during the scope of the project.
Bayer supports seriously ill people during their treatment with a variety of active substances and prescription medication worldwide. Longtime and wellknown pharmaceuticals are constantly complemented with new developments, especially in the cardiology field, women’s health, oncology, hematology and ophthalmology. Therapeutic areas in which the medical needs are still not met are a focal point in drug discovery. One of these areas is the biologics field that offers improved treatment approaches for different diseases. New production capacities for the freeze drying and filling of these substances are currently being developed in Leverkusen, one of the largest production locations for Pharmaceuticals. The production line for parenteral pharmaceuticals is located in an existing production building. The entire line was delivered by Optima Pharma, fully automated and linked to the supervisory level central Bayer IT system MES and Historian via SCADA programming – a very impressive turnkey project.
It is not only impressive that Optima Pharma delivered both freeze dryers, the vial filler consisting of loading and unloading systems, isolator, washer, tunnel, filler, capper, tray systems, and the appropriate CIP system – all from one source. But Optima Pharma Mornshausen was also in charge of the automation especially for the generation of the SCADA application and all required network systems and components. André Schmidt, project supervisor of automation, says: “A SCADA project of this magnitude was a first for us. The biggest challenge was to design the interfaces to the customer’s systems.”
In order to establish the required automation including the integration into the “Bayer IT world” the Optima engineers used the Siemens process control system PCS 7, version 8.1 with Simatic Batch for the freeze dryer management and Simatic WinCC with PM-Quality for the filling line. The complete SCADA application includes the visualization and operation of the line, as well as the query for GMP related data, batch reports and the recipe and batch management. The SCADA system transfers order data and target values to the PLCs of the individual machines, creating a paperless production where the operator only has to intervene if a fault occurs.
ANTICIPATING THE FDA RECOMMENDATION
However, Bayer wanted even more: they had very strict requirements regarding the IT security, especially the data integrity. At first, the FDA draft guidance “Data Integrity and Compliance with cGMP”, published in 2016, was only a suggestion. However, the project managers at Bayer and Optima Pharma wanted to guarantee the data integrity with the new production line at all times. There- fore, some requirements were just determined after the actual engineering phase. IT-security was becoming an increasingly high priority during the project. The access protection of the servers for the line management was a focal point since large industrial companies, such as Bayer, are suspect to thousands of hacking attacks every year.
Daniel Keck, SCADA programmer at Optima Pharma and sub-project manager, explained: “Ensuring IT security of individual, autonomous machines is quite easy. The new line however, is integrated into the Bayer network. Theoretically, it is possible with the internet to access the system from any part of the world, down to the machine level plant bus.
To prevent manipulation and vandalism, the entire line is shielded with a firewall-cluster usually only used for the oil and gas industry or large-scale commodity productions like the chemical industry. The user management system is also unlike any other system for production machinery. Passwords, user names and groups are centrally managed in the global Bayer system. Service users are only created as an exception in the local system in Leverkusen and is intensely regulated and limited. Login-requests are transferred from the operator panel of the machine with a LogOn server to the Bayer system. The user management in WinCC only assigns the access rights to the user. “This ensures complete traceability of all mandatory audit trails of manipulations by individual people,” ensured Daniel Keck. He added: “A complete white listing, using a separate server, prohibits unknown processes from occurring on the server.” After all, seriously ill people are depending on the product that will be filled on this line. Therefore, everything has to be done to avoid disruptions in production.
The implementation of the necessary data integrity and IT security was an important step in the SCADA project for Bayer. In addition, the SCADA system had to be according to Bayer’s requirements. Transparency and operator friendliness were the focal point. Since the entire production line has to be paperless, the SCADA also has to record alarms, audit trails, measurements, batch data and historical trends and generate the batch reports. The conformity with GAMP5 and FDA CFR Part 11 is guaranteed, this was business as usual for Daniel Keck and his 35 coworkers that worked on the Bayer project.
André Schmidt, Optima Pharma
OPC CONNECTION TO MES AND HISTORIAN
“The development of the start-stop logic to the Bayer MES was tricky”, explained Daniel Keck, “since there was no standard protocol. As for all special requirements, we used PCS7 as the programming environment.”
Siemens, as a sub-vendor, supplied the firewalls and integration layer. “It is due to the great cooperation of Siemens, Bayer and Optima Pharma that everything works so smoothly,” said Daniel Keck. Optima Pharma is not only responsible for the machinery, server, network technology, operator stations and complete automation of the line but also the entire qualification. In order to include the expertise of the Bayer system operators, the IQ (Installation Qualification) and OQ (Operational Qualification) took place in close collaboration with Bayer. “Each item was discussed with the person in charge at Bayer and then implemented by us,” explained Daniel Keck.
The FAT at Optima Pharma was completed in December, 2016 therefore, some parts of the line were sent to Bayer at the beginning of 2017. The additional implementation will require some extra time, due to policies regarding the IT security that just emerged during the project progression. Many of the customer’s requirements can only be completed on-site. Additional server and safety structures were needed and had to be scheduled.
“All mandatory audit trail manipulations
are completely traceable. And a whitelisting
prevents unknown interventions to
avoid production interruptions.”
Daniel Keck, SCADA programmer reports
SECURING VALUABLE DATA FOR OEE
Additional subjects such as the data transfer that are not a priority were transferred into the systems with subsequent customer requirements and implemented by the Optima programmers. Therefore, data sets are not collected in the machine that generates the data but are stored centrally. The data can be used for various statistical analysis by Bayer, for example to improve the OEE (Overall Equipment Effectiveness).
The preparation for the qualification phase started in September, 2017 and the actual qualification, starting with the IQ, began early in January, 2018. This phase will be completed in June, 2018 and followed by the OQ. Daniel Keck is still involved in the project process. The sub-project manager studied systems engineering and technical cybernetics before he started with Optima Pharma. “I did everything that was necessary onsite for this project, even being the emergency locksmith. Therefore, I learned a lot in a very short amount of time,” he said with a smile. Now he can proudly look back: a complex pharmaceutical line, integrated into the subordinate customer system, with extremely high safety standards and a process control system that will fulfill the FDA requirements for sure.
André Schmidt who offered the opportunity to the young SCADA programmer does not regret his decision: “We showed our automation capabilities with the Bayer line.” This project already led to multiple subsequent orders, with some complex and some not as complex re- quirements as the Bayer Leverkusen order. “Our data integrity expertise is very helpful. The pharmaceutical world is waking up; this subject is becoming increasingly important,” said André Schmidt. Therefore, the next Optima Pharma line will be perfectly prepared for paperless production according to the latest FDA and GAMP requirements.
André Schmidt, Supervisor Automation at Optima Pharma