The Many Faces of Product R&D: The Test Manager

Original

Traditionally, the Test Manager serves as the lead of the integration or system testing team, focusing primarily on testing software, controlling quality, and ensuring products meet standards before release, acting as a gatekeeper of quality. However, in agile development models, this role has evolved into that of a builder of the overall quality system. Their responsibilities are no longer confined to defect detection but extend to systematically enhancing the team's testing efficiency and product quality through process optimization, resource coordination, and technical advocacy.

I. Workflow and Core Tasks

1. Traditional vs. Agile Testing Processes

Traditional Testing Process

In traditional software development, most teams include a dedicated testing unit. In formal processes, the Product Manager first drafts the Functional Requirements Specification (FRS). The development team then builds accordingly, while the testing team writes and executes test cases based on the FRS. When defects are identified, they are logged into a system. The development team reproduces, locates, and fixes the issues, and the testing team verifies the fixes before closing the defects. Defects are typically categorized into four or five severity levels, and no major or critical unresolved defects are allowed before product release. In informal processes, frequent requirement changes occur without a dedicated FRS, and testing often relies on manual, simplified checks before delivery to the customer.

Agile Testing Process

Compared to traditional waterfall or iterative development models, agile development shortens the feedback cycle from requirements to testing, aligning with modern software's need for rapid response to changes. In this model, some system testing tasks, primarily functional testing of features developed by the development team, are shifted left to the development team, enabling quick feedback loops. However, the system testing team is not eliminated; its role evolves to ensure comprehensive product quality. Tasks such as stress testing, project scenario testing, version compatibility testing, and security testing are difficult for development teams to handle alone and still require dedicated testing teams. This is especially true for testing industrial control products, which involve both hardware and software. Setting up test environments that simulate real-world conditions is technically complex and costly, often leading companies to centralize resources in specialized testing centers.

That said, shifting testing left also introduces challenges. Previously, functional testing was centralized within the testing team, making it easier to control test coverage. Now, with functional testing moved to development teams, whose primary goal is feature delivery, testing quality may be compromised under tight deadlines. Therefore, the Test Manager must monitor the test coverage and quality across development teams.

2. Test Coverage and Product Release

"Complete test pass" is an impractical goal, as testing can theoretically continue indefinitely, particularly in error scenario testing. Even with extended testing periods, new defects may still be discovered. While more testing generally leads to better product quality, it also increases costs and delays delivery. Therefore, determining the appropriate scope and depth of testing to achieve optimal R&D cost-effectiveness is a core competency of the Test Manager, with test coverage serving as the key metric.

In practice, functional testing should achieve 100% positive scenario coverage, while negative testing should cover most common error scenarios. Performance, security, stress, and other specialized tests remain the responsibility of the system testing team, and the Test Manager must develop plans to ensure their coverage. Before product release, all types of test coverage must meet established standards. Once standards are met and test execution is complete, the regression phase begins. During this phase, no new test cases are executed, and efforts focus on defect resolution. When all critical and major defects are resolved, the regression period concludes, and the Test Manager submits a report to initiate the release process. After release, the development team shifts focus to the next version while also addressing field defects and releasing official patches. These patches must pass full regression testing, and the Test Manager is responsible for ensuring their quality.

3. Building a Quality Assurance System

Manually inspecting the test coverage of each development team is impractical. The Test Manager must establish a quality assurance system that includes various safeguards beyond coverage checks. Regarding test coverage verification, existing tools can only inspect at the code level and cannot cover user story coverage. Therefore, it is stipulated that when Product Owners (POs) break down user stories, they must simultaneously define test cases, and the granularity of user stories should align with functional test cases. This enables the system to automatically track functional test coverage and execution status. Implementing this rule depends on the Test Manager building an automated system and continuously promoting its adoption. Additionally, the system incorporates check-in rule verification, automatically inspecting compliance during code check-in and rejecting non-compliant code. Regular static code checks are also conducted, and critical defects identified by static code analysis must be resolved before product release.

4. Test Framework Construction

High-coverage automated testing is fundamental to modern software development, with manual testing reserved only for specific scenarios. Most internet products implement CI/CD (Continuous Integration/Continuous Deployment), where code is automatically integrated and system-tested upon check-in, and released if tests pass. For industrial products involving both hardware and software, complete automation may not be feasible, but release cycles can still be shortened. Development teams typically build their own unit testing frameworks, as business differences make standardization unnecessary. However, during integration and system testing phases, the Test Manager must establish a unified testing framework covering common basic functions. For example, a PLC integration testing framework should support automatic program generation and provide compilation invocation interfaces, while an integration testing framework should enable automatic screen generation and point binding. A unified framework significantly reduces the cost and time required for test case development.

II. Competency Requirements for an Excellent Test Manager

1. Professional Integrity

Within the R&D system, the Test Manager often occupies a challenging position where efforts may go unappreciated. Whether quality standards are enforced too leniently or too strictly, dissatisfaction can arise. However, an excellent Test Manager must demonstrate unwavering commitment and persistence regarding product quality, as they represent the final defense line before a product's release. This is particularly critical for industrial products, where quality issues could lead to operational downtime, or even catastrophic equipment damage and loss of life. In one documented case, just before the scheduled release of an industrial software, testing revealed an intermittent system crash. Senior management proposed downgrading the issue's severity to proceed with the release, but the Test Manager resisted the pressure and refused to sign off. Ultimately, the problem was resolved, ensuring product quality.

2. Technical Capabilities

A Test Manager must possess a comprehensive understanding of testing technology systems. Traditional skills include proficiency with various testing tools, code inspection tools, and specialized testing methodologies for functionality, performance, security, and other areas. In the agile era, they also need the ability to design automated testing frameworks and integrate automated tests into CI/CD pipelines, enabling automatic validation upon code submission. In the AI era, Test Managers must learn to leverage AI tools, such as using AI visual testing tools like Testim, generating test cases with DeepSeek, and employing machine learning models to predict defects. These tools can increase test coverage to 80-90% while reducing testing costs. Simultaneously, they need to understand AI testing methods like adversarial testing and metamorphic testing to address false positive issues in AI models.

3. Management and Communication Skills

As a technical management role, the Test Manager must balance technical and managerial responsibilities. In team management, they lead the testing team to execute plans and monitor progress. In project management, they analyze testing requirements, propose design solutions, develop testing programs, provide results and recommendations, create work plans, allocate personnel appropriately, schedule timelines, and ensure high-quality project delivery.

With testing technologies evolving rapidly in the AI era, Test Managers must regularly organize technical training to enhance team capabilities. Furthermore, communication skills are crucial. They need to coordinate with Product Owners on test coverage and framework establishment, discuss integration test cases and defect classification with Product Managers, consult with Architects on automated testing frameworks and testable interface design, and report release schedules and quality assurance status to senior management.

4. Stress Resilience

Test Managers face multiple pressures, including tight product release deadlines and scrutiny when field issues arise, often facing questions like "why wasn't this defect discovered in the lab?" Only those with strong stress resilience can competently fulfill this role.

The Test Manager plays a crucial role in product quality assurance, not only guarding the final quality checkpoint before product release and acting as the gatekeeper in product R&D, but also significantly influencing the enterprise's innovation capability and market competitiveness. This position is indispensable within the product R&D system.